I have been working heavily in C# CE (Compact Edition) v2.0 for the last 2 years for clients, and one of the very many things that I was never really happy with (in at least that version of the language, though it looks like it might plague all versions of C#) is the available thread synchronization tools. I’ve come to love the lock/wait/notify model (in Java it’s synchronized/wait/notify and in Perl it’s lock/cond_wait/cond_signal), but I have found nothing as intuitive and safe to use in C#. To alleviate this, I went ahead and wrote my own ThreadLockAndWait class that achieves this functionality.

• Lock is not required before CondSignal (it does its own inner lock and unlock)
• If ReacquireLockAfterWait is false, in which case CondWait will not lock again after signaled and just continue immediately
• Only 1 thread can be CondWaiting at a time (If one is CondWaiting and is signaled but not reacquired the lock, its ok for another to start CondWaiting)

public class ThreadLockAndWait
{
	private Mutex TheLock=new Mutex(), CondWaitLock=new Mutex(); //CondWaitLock makes sure 1 thread stops waiting before the next one starts waiting
	private ManualResetEvent WaitTimer=new ManualResetEvent(false);
	private string OwnersThreadName=null;
	private int OwnerLockCount=0;

	public void Lock()
	{
		TheLock.WaitOne();
		if(OwnerLockCount++==0)
			OwnersThreadName=Thread.CurrentThread.Name;
	}
	public void UnLock()
	{
		TheLock.WaitOne();
		if(OwnerLockCount==0)
		{
			TheLock.ReleaseMutex();
			throw new Exception("Cannot unlock if not locked");
		}
		TheLock.ReleaseMutex();
		if(--OwnerLockCount==0)
			OwnersThreadName=null;
		TheLock.ReleaseMutex();
	}
	public void CondWait() { RealCondWait(-1, true); }
	public void CondWait(bool ReacquireLockAfterWait) { RealCondWait(-1, ReacquireLockAfterWait); }
	public void CondTimedWait(int TimeToWait) { RealCondWait(Math.Max(0, TimeToWait), true); }
	public void CondTimedWait(int TimeToWait, bool ReacquireLockAfterWait) { RealCondWait(Math.Max(0, TimeToWait), ReacquireLockAfterWait); }
	private void RealCondWait(int TimeToWait, bool ReacquireLockAfterWait)
	{
		//Prepare to wait
		TheLock.WaitOne();
		if(OwnerLockCount==0)
		{
			TheLock.ReleaseMutex();
			throw new Exception("Cannot wait if not locked");
		}
		CondWaitLock.WaitOne(); //Release this wait before the next one starts
		WaitTimer.Reset();
		TheLock.ReleaseMutex();

		//Release all locks
		int PreviousLockCount=OwnerLockCount;
		OwnersThreadName=null;
		OwnerLockCount=0;
		if(PreviousLockCount!=1)
			System.Diagnostics.Debug.Print("Warning, mutex has multiple locks from thread!");
		for(int i=0;i<PreviousLockCount;i++)
			TheLock.ReleaseMutex();

		//Wait
		if(TimeToWait>0)
			WaitTimer.WaitOne(TimeToWait, false);
		else if(TimeToWait!=0)
			WaitTimer.WaitOne();
		CondWaitLock.ReleaseMutex();

		//Reacquire lock
		if(!ReacquireLockAfterWait)
			return;
		for(int i=0;i<PreviousLockCount;i++)
			TheLock.WaitOne();
		OwnerLockCount=PreviousLockCount;
		OwnersThreadName=Thread.CurrentThread.Name;
	}

	public void CondSignal()
	{
		TheLock.WaitOne();
		WaitTimer.Set();
		TheLock.ReleaseMutex();
	}
}

I was having problems on one of our production Linux cPanel servers in which our backup drive was not able to hold all the data from our primary drive for both our daily and weekly backups. An easy hack to fix this is to mount any subfolders you wish to exclude (generally very large ones) as a readonly temp file system in the appropriate backup folder. With this method, you can selectively exclude individual directories to one or more of the daily/weekly/monthly backup folders.

The only downside to this method is that pkgacct (called by cpbackup) logs will throw readonly file system errors for each file that cannot be copied.

rm -rf PATH;
mkdir -p PATH;

mount tmpfs PATH -t tmpfs -o defaults,ro

tmpfs PATH tmpfs defaults,ro 0 0

cPanel also recently added (experimental) hard linking for backups, which really helps out with space concerns, and makes the need for this script a bit less.

I wanted to play around with Google's go language a little so I ended up decided on making a simple class that helps create a TCP connection between a server and client that is encrypted via TLS, or not, depending upon a flag. Having the ability to not encrypt a connection is useful for debugging and testing purposes, especially if other people are needing to create clients to connect to your server.

The example server.go file listens on port 16001 and for every set of data it receives, it sends the reversed string back to the client. (Note there are limitations to the string lengths in the examples due to buffer and packet payload length restrictions).

The example client.go file connects to the server (given via the 1st command line parameter), optionally encrypts the connection (depending upon the 2nd command line parameter), and sends the rest of the parameters to the server as strings.

• StartServer: Goes into a connection accepting loop. Whenever a connection is accepted, it checks the data stream for either the "ENCR" or "PTXT" flags, which control whether a TLS connection is created or not. The passed "clientHandler" function is called once the connection is completed.
• StartClient: Connects to a server, passes either the "ENCR" or "PTXT" flag as noted above, and returns the finished connection.

Connections are returned as "ReadWriteClose" interfaces. Creating the pem and key certificate files is done via openssl. You can just google for examples.

package main
import ( "./encryptedtcp"; "fmt"; "log" )

func main() {
	if err := encryptedtcp.StartServer("server.pem", "server.key", "0.0.0.0:16001", handleClient); err != nil {
		log.Printf("%q\n", err) }
}

func handleClient(conn encryptedtcp.ReadWriteClose) {
	buf := make([]byte, 512)
	for {
		//Read data
		n, err := conn.Read(buf)
		if err != nil {
			log.Printf("Error Reading: %q\n", err); break }
		fmt.Printf("Received: %q\n", string(buf[:n]))

		//Reverse data
		for i, m := 0, n/2; i<m; i++ { //Iterate over half the list
			buf[i], buf[n-i-1] = buf[n-i-1], buf[i] } //Swap first and half of list 1 char at a time

		//Echo back reversed data
		n, err = conn.Write(buf[:n])
		if err != nil {
			log.Printf("Error Writing: %q\n", err); break }
		fmt.Printf("Sent: %q\n", string(buf[:n]))
	}
}

package main
import ( "./encryptedtcp"; "fmt"; "log"; "os" )

func main() {
	//Confirm parameters, and if invalid, print the help
	if len(os.Args) < 4 || (os.Args[2] != "y" && os.Args[2] != "n") {
		log.Print("First Parameter: ip address to connect to\nSecond Parameter: y = encrypted, n = unencrypted\nAdditional Parameters (at least 1 required): messages to send\n"); return }

	//Initialize the connection
	conn, err := encryptedtcp.StartClient("client.pem", "client.key", os.Args[1]+":16001", os.Args[2]=="y" )
	if err != nil {
		log.Printf("%q\n", err); return }
	defer conn.Close()

	//Process all parameters past the first
	buf := make([]byte, 512)
	for _, msg := range os.Args[3:] {
		//Send the parameter
		if(len(msg)==0) {
			continue }
		n, err := conn.Write([]byte(msg))
		if err != nil {
			log.Printf("Error Writing: %q\n", err); break }
		fmt.Printf("Sent: %q\n", msg[:n])

		//Receive the reply
		n, err = conn.Read(buf)
		if err != nil {
			log.Printf("Error Reading: %q\n", err); break }
		fmt.Printf("Received: %q\n", string(buf[:n]))
	}
}

//A simple TCP client/server that can be encrypted (via tls) or not, depending on a flag passed from the client

package encryptedtcp

import ( "crypto/rand"; "crypto/tls"; "net"; "log" )

//Goes into a loop to accept clients. Returns a string on error
func StartServer(certFile, keyFile, listenOn string, clientHandler func(ReadWriteClose)) (error) {
	//Configure the certificate information
	cert, err := tls.LoadX509KeyPair(certFile, keyFile)
	if err != nil {
		return MyError{"Cannot Load Keys", err} }
	conf := tls.Config{Certificates:[]tls.Certificate{cert}, ClientAuth:tls.RequireAnyClientCert, Rand:rand.Reader}

	//Create the listener
	listener, err := net.Listen("tcp", listenOn)
	if err != nil {
		return MyError{"Cannot Listen", err} }
	defer listener.Close()

	//Listen and dispatch clients
	for {
		conn, err := listener.Accept()
		if err != nil {
			return MyError{"Cannot Accept Client", err} }
		go startHandleClient(conn, &conf, clientHandler)
	}

	//No error to return - This state is unreachable in the current library
	return nil
}

//Return the io stream for the connected client
func startHandleClient(conn net.Conn, conf* tls.Config, clientHandler func(ReadWriteClose)) {
	defer conn.Close()

	//Confirm encrypted connection flag (ENCR = yes, PTXT = no)
	isEncrypted := make([]byte, 4)
	amountRead, err := conn.Read(isEncrypted)
	if err != nil {
		log.Printf("Cannot get Encrypted Flag: %q\n", err); return }
	if amountRead != 4 {
		log.Printf("Cannot get Encrypted Flag: %q\n", "Invalid flag length"); return }
	if string(isEncrypted) == "PTXT" { //If plain text, just pass the net.Conn object to the client handler
		clientHandler(conn); return
	} else if string(isEncrypted) != "ENCR" { //If not a valid flag value
		log.Printf("Invalid flag value: %q\n", isEncrypted); return }

	//Initialize the tls session
	tlsconn := tls.Server(conn, conf)
	defer tlsconn.Close()
	if err := tlsconn.Handshake(); err != nil {
		log.Printf("TLS handshake failed: %q\n", err); return }

	//Pass the tls.Conn object to the client handler
	clientHandler(tlsconn)
}

//Start a client connection
func StartClient(certFile, keyFile, connectTo string, isEncrypted bool) (ReadWriteClose, error) {
	//Configure the certificate information
	cert, err := tls.LoadX509KeyPair(certFile, keyFile)
	if err != nil {
		return nil, MyError{"Cannot Load Keys", err} }
	conf := tls.Config{Certificates:[]tls.Certificate{cert}, InsecureSkipVerify:true}

	//Connect to the server
	tcpconn, err := net.Dial("tcp", connectTo)
	if err != nil {
		return nil, MyError{"Cannot Connect", err} }

	//Handle unencrypted connections
	if !isEncrypted {
		tcpconn.Write([]byte("PTXT"))
		return tcpconn, nil //Return the base tcp connection
	}

	//Initialize encrypted connections
	tcpconn.Write([]byte("ENCR"))
	conn := tls.Client(tcpconn, &conf)
	conn.Handshake()

	//Confirm handshake was successful
	state := conn.ConnectionState()
	if !state.HandshakeComplete || !state.NegotiatedProtocolIsMutual {
		conn.Close()
		if !state.HandshakeComplete {
			return nil, MyError{"Handshake did not complete successfully", nil}
		} else {
			return nil, MyError{"Negotiated Protocol Is Not Mutual", nil} }
	}

	//Return the tls connection
	return conn, nil
}

//Error handling
type MyError struct {
	Context string
	TheError error
}
func (e MyError) Error() string {
	return e.Context+": "+e.TheError.Error(); }

//Interface for socket objects (read, write, close)
type ReadWriteClose interface {
	Read(b []byte) (n int, err error)
	Write(b []byte) (n int, err error)
	Close() error
}

Rsync is a spectacular bash utility for doing file copying and/or syncing operations. It has a multitude of switches to help optimize and handle any requirements for file copy operation over a local computer or network. However, sometimes networks are less than stable and stalls can happen during an rsync (or scp). This is quite the nuisance when doing very large (i.e. GB) transfers. To solve this, the following script can be used to auto resume a stalled rsync.

export Result=1;
while [ $Result -ne 0 ]; do
  echo "STARTING ($Result) @" `date`;
  rsync -Pza --timeout=10 COPY_FROM COPY_TO_USER@COPY_TO_HOST:COPY_TO_LOCATION;
  Result=$?;
  sleep 1;
done

• The -P switch is highly suggested as it activates:
  • --partial: This keeps a file even if it doesn’t finish transferring so it can be resumed when rsync is restarted. This is especially important if you have very large files.
  • --progress: This shows you the progress of the current file copy operation.
• The -z switch turns on gzip compression during the file transfer, so may only be useful depending on the circumstances.
• The -a switch stands for “archive” and is generally a good idea to use. It includes the switches:
  • -r: Recurse into folders
  • -t: Preserves file modification time stamp. This is highly recommended for this, and incremental backups, as rsync, by default, skips files whose file sizes and modification times match.
  • -l and -D: Preserve file type (i.e. symlinks)
  • -p: Preserve file (chmod) permissions
  • -g and -o: Preserve file owners.
• The --timeout is the crux of the script, in that if an I/O timeout of 10 seconds occurs, the rsync exits prematurely so it can be restarted.

For more useful switches and information, see the rsync man page.

export Result=1; #This will hold the result of the rsync. Set to 1 so the first loop check will fail.
while [ $Result -ne 0 ]; do #Loop until rsync result is successful
  echo "STARTING ($Result) @" `date`; #Inform the user of the time an rsync is starting and the last rsync failure code
  rsync -Pza --timeout=10 COPY_FROM COPY_TO_USER@COPY_TO_HOST:COPY_TO_LOCATION; #See rest of post for switch information
  Result=$?; #Store the result of the rsync
  sleep 1; #This is an optional 1 second timeout between attempts
done

I had the need today to be able to set the current time zone for an application in multiple computer languages by the hourly offset from GMT/UTC, which turned out to be a lot harder than I expected. It seems most time zone related functions, at least in Linux, expect you to use full location strings to set the current time zone offset (i.e. America/Chicago).

After a lot of research and experimenting, I came up with the following results. All of these are confirmed working in Linux, and most or all of them should work in Windows too.

#include <stdio.h> //snprintf
#include <stdlib.h> //setenv, atoi
#include <time.h> //tzset

...

char Buffer[10];
snprintf(Buffer, 10, "GMT%i", -atoi(TimeZone));
setenv("TZ", Buffer, 1);
tzset();
		

use POSIX qw/tzset/;
$ENV{TZ}='GMT'.(-$TimeZone);
tzset;
		

$Query='SET time_zone="'.($TimeZone>=0 ? '+' : '').$TimeZone.':00"';
		

date_default_timezone_set('Etc/GMT'.($TimeZone<=0 ? '+' : '').(-$TimeZone));
		

Since I found out that NTFS now has semi-working native symlinks, I have updated the symlinking script used in the Combining an Android Project's Versions post. This script creates relative symlinks now through Perl instead of absolute hard links through Bash. It is as follows:

#!/usr/bin/perl
#Run this file to install links to shared files into all branches
use warnings;
use strict;

#Configuration
my $SharedDirectoryName="Shared";
my $NonProjectDirectories="^\\.(|/\\.git|/$SharedDirectoryName)\$"; #Non Project directories (., .git, $SharedDirectoryName)
my $IsWindows=(index(lc(`uname`), 'cygwin')!=-1);

#Create a symlink
sub MakeLink
{
	my ($LinkTarget, $LinkName, $IsWindows, $IsDirectory)=@_;

	#Create the target directory if it does not exist
	my $LinkDirectory=$LinkName;
	$LinkDirectory =~ s/\/[^\/]+$//;
	if(!-e $LinkDirectory) {
		print "Creating directory: $LinkDirectory\n";
		`mkdir -p "$LinkDirectory"`;
	}
	
	#If the link already exists, issue a warning
	if(-l $LinkName) {
		print "Link already exists: $LinkName\n";
		return;
	}

	#Create the relative symlink
	my $RelativePathFromLinkToTarget=('../' x ($LinkName =~ tr/\///)).$LinkTarget; #Determine the relative path between the link and its target
	my $Command;
	if(!$IsWindows) { #Create the Linux command
		$Command="ln -s \"$RelativePathFromLinkToTarget\" \"$LinkName\"";
	}
	else #Create the Windows command
	{
		#Replace /s in path with \s
		$RelativePathFromLinkToTarget =~ s/\//\\/g;
		$LinkName =~ s/\//\\/g;
		
		$Command='cmd /c mklink'.($IsDirectory ? ' /d' : '')." \"$LinkName\" \"$RelativePathFromLinkToTarget\"";
	}

	print "$Command\n";
	`$Command`;
}

#Find required information from file searches
my @LocalBranches=grep(!/$NonProjectDirectories/, `find -maxdepth 1 -type d`); #Find version folders by ignoring Non Project directories
my @Files=split(/\n?^$SharedDirectoryName\//m, substr(`find $SharedDirectoryName -type f`, 0, -1)); shift @Files; #Find shared files

#Propagate shared files into different versions
foreach my $LocalBranch (@LocalBranches) {
	$LocalBranch=substr($LocalBranch, 2, -1); #Remove ./ and new line separator
	foreach my $File (@Files) {
		MakeLink("$SharedDirectoryName/$File", "$LocalBranch/$File", $IsWindows, 0);
	}
}

I got tired a long time ago of losing what I was currently working on in SSH sessions when they were lost due to disconnects from network connectivity issues. To combat this I have been using screen when running sessions that I can absolutely not lose, but the problem still persists in other sessions or when I forgot to run it. The easy solution to this was to add screen to one of my bash init scripts (~/.bashrc [or ~/.bash_profile]) as follows:

alias autoscreen="screen -x -RR && exit"
if [[ "$TERM" == cygwin* || "$TERM" == xterm* ]]; then autoscreen; fi

if [ $TERM == "xterm" ]; then screen -R pts-0.`hostname` && exit; fi

• The main screen command is now an “alias” to help out with some bash parsing problems.
• The resume parameters are now “-x -RR” which first attempts to multiplex a session, and if that fails, it creates a session. With multiplexing turned on, everyone uses the same screen session and can interact with each other, and you don’t have to worry about accidentally connecting to the wrong screen session or having new ones turned on. The only problem is sometimes you may accidentally step on other user’s toes :-)
• The special screen session name was removed so it always starts with the default name (easier to manually interact with)
• I added detection of multiple term names (cygwin and xterm), and added a wildcard at the end of each since there is often suffixes to these names. More term names can easily be added using this syntax.

Edit on 2010-12-30 @ 3:50am: I changed != "screen" to == "xterm" because otherwise scp and some other non-term programs were failing. You may have to use something other than “xterm” for your systems.

Edit on 2010-1-24 @ 2:00pm: I added the exit; so the terminal automatically exits when the screen session closes.

For a number of years now when on insecure network connections I have been routing my computer to the Internet through secure tunnels and VPNs, but I’ve been interested in trying out different types of VPN software lately so I can more easily help secure friends who ask of it. This would mainly include ease of installation and enabling, which partly requires no extra software for them to install.

Unfortunately, Windows 7 and Android (and probably most other software) only support PPTP and L2TP/IPSEC out of the box. While these protocols are good for what they do, everything I have read says OpenVPN is superior to them as a protocol. I was very frustrated to find out how little support OpenVPN actually has today as a standard in the industry, which is to say, you have to use third party clients and it is rarely, if ever, included by default in OSes. The OpenVPN client and server aren’t exactly the easiest to set up either for novices.

So on to the real point of this post. The sample client and server configurations for OpenVPN were set up just how I needed them except they did not include two important options for me: User authentication and full client Internet forwarding/tunneling/gateway routing. Here is how to enable both.

• Add the following options to the server configuration file:
  • push "redirect-gateway def1" #Tells the client to use the server as its default gateway
  • push "dhcp-option DNS 10.8.0.1" #Tells the client to use the server as its DNS Server (DNS Server's IP address dependent on configuration)
• Run the following commands in bash:
  • iptables -t nat -A POSTROUTING -s 10.8.0.0/24 -o eth0 -j MASQUERADE #This command assumes that the VPN subnet is 10.8.0.0/24 (taken from the server directive in the OpenVPN server configuration) and that the local ethernet interface is eth0.
  • echo '1' > /proc/sys/net/ipv4/ip_forward #Enable IP Forwarding (This one is not mentioned in the OpenVPN howto)

To set up username/password authentication on the server, an authorization script is needed that receives the username/password and returns whether the login information was successful (0) or failed (1). The steps to set up this process are as follows:

• Add the following options to the server configuration file:
  • auth-user-pass-verify verify.php via-env #The third argument (method) specifies whether to send the username and password through either a temporary file (via-file) or environment variables (via-env)
  • script-security 3 system #Allows OpenVPN to run user scripts and executables and send password authentication information through environment variables. While "system" is deprecated, I had to use it or external commands like ifconfig and route were failing with "failed: could not execute external program"
• Add the following options to the client configuration file:
  • auth-user-pass #Request user credentials to log in
• The final step is to create the verify.php (see auth-user-pass-verify configuration above) script which returns whether it was successful, and also outputs its success to stdout, which is added to the OpenVPN log file.

  #!/usr/bin/php -q
  <?
  //Configuration
  $ValidUserFile='users.txt'; //This file must be in htpasswd SHA1 format (htpasswd -s)
  $Method='via-env'; //via-file or via-env (see auth-user-pass-verify configuration above for more information)
  
  //Get the login info
  if($Method=='via-file') //via-file method
  {
  	$LoginInfoFile=trim(file_get_contents('php://stdin')); //Get the file that contains the passed login info from stdin
  	$LoginInfo=file_get_contents($LoginInfoFile); //Get the passed login info
  	file_put_contents($LoginInfoFile, str_repeat('x', strlen($LoginInfo))); //Shred the login info file
  	$LoginInfo=explode("\n", $LoginInfo); //Split into [Username, Password]
  	$UserName=$LoginInfo[0];
  	$Password=$LoginInfo[1];
  }
  else //via-env method
  {
  	$UserName=$_ENV['username'];
  	$Password=$_ENV['password'];
  }
  
  //Test the login info against the valid user file
  $UserLine="$UserName:{SHA}".base64_encode(sha1($Password, TRUE)); //Compile what the user line should look like
  foreach(file($ValidUserFile, FILE_IGNORE_NEW_LINES) as $Line) //Attempt to match against each line in the file
  	if($UserLine==$Line) //If credentials match, return success
  	{
  		print "Logged in: $UserName\n";
  		exit(0);
  	}
  
  //Return failure
  print "NOT Logged in: $UserName\n";
  exit(1);
  ?>
  		

The following is a Python script that automatically pings a requested web address at a given interval. It was made as a quick favor for a friend. Here is the downloadable source code and Windows binary.

1. The URL to ping (The default pings the GetIP script)
2. The millisecond interval between pings (Default=600000=10 minutes)

#!python
from sys import stderr
from urllib import urlopen
from time import localtime, strftime, sleep

#Main function
def Main():
    #Open the settings file
    SettingFileName='PingURL.cfg';
    Settings=[] #Blank out settings file variable in case it can't be opened
    try:
        Settings=open(SettingFileName, 'r').readlines()
    except IOError as (errno, strerror):
        stderr.write('Cannot open {0} configuration file: I/O error({1}): {2}\n'.format(SettingFileName, errno, strerror))
        return

    #Confirm valid settings were passed
    if(len(Settings)<2):
        stderr.write('Not enough settings found in settings file\n')
        return
    try:
        IntervalTime=int(Settings[1])
    except:
        stderr.write('Invalid interval time\n')
        return
        
    #Ping the URL indefinitely
    while(True):
        try:
            URLText=urlopen(Settings[0]).read()
        except:
            URLText='READ FAILED'
        print 'URL Pinged At {0}: {1}'.format(strftime('%Y-%m-%d %H:%M:%S', localtime()), URLText)
        sleep(IntervalTime/1000)

#Run the program
Main()

I often find SSH tunnels absolutely indispensable in my line of work for multiple reasons including secure proxies (tunneling) over insecure connections and connecting computers and programs together over difficult network setups involving NATs and firewalls.

One such example of this I ran into recently is that I have a server machine (hereby called “the client”) that I wanted to make sure I have access to no matter where it is. For this I created an auto initializing SSH remote port tunnel to a server with a static IP Address (hereby called “the proxy server”) which attempts to keep itself open when there is problems.

The first step of this was to create the following bash script on the client that utilizes the OpenSSH’s client to connect to an OpenSSH server on the proxy server for tunneling:

#!/bin/bash
for ((;;)) #Infinite loop to keep the tunnel open
do
	ssh USERNAME@PROXYSERVER -o ExitOnForwardFailure=yes -o TCPKeepAlive=yes -o ServerAliveCountMax=2 -o ServerAliveInterval=10 -N -R PROXYPORT:localhost:22 &>> TUNNEL_LOG #Create the SSH tunnel
	echo "Restarting: " `date` >> TUNNEL_LOG #Write to the log file "TUNNEL_LOG" whenever a restart is attempted
	sleep 1 #Wait 1 second in between connection attempts
done

For this to work, you will need to set up public key authentication between the client and utilized user on the proxy server. To do this, you have to run “ssh-keygen” on the client. When it has finished, you must copy the contents of your public key file (most likely at “~/.ssh/id_rsa.pub”) to the “~/.ssh/authorized_keys” file on the user account you are logging into on the proxy server. You will also have to log into this account at least once from the client so the proxy server’s information is in the client’s “known_hosts” file.

For security reasons you may want the user on the proxy server to have a nologin shell set since it is only being used for tunneling, and the above mentioned public key will allow access without a password.

For network access reasons, it might also be worth it to have the proxy port and the ssh port on the proxy server set to commonly accessible ports on all network setups (that a firewall wouldn’t block). Or you may NOT want to have it on common ports for other security reasons :-).

If you want the proxy port on the proxy server accessible from other computers (not only the proxy server), you will have to turn on “GatewayPorts” (set to “yes”) in the proxy server’s sshd config, most likely located at “/etc/ssh/sshd_config”.

The next step is to create a daemon that calls this script. The normal method for this in Linux would be to use inittab. This can be a bit cumbersome though with Linux distributions that use upstart, like Ubuntu, so I just cheated in it and created the following script to initialize a daemon through rc.d:

#!/bin/bash
echo -e '#!/bin/bash\nfor ((;;))\ndo\n  ssh USERNAME@PROXYSERVER -o TCPKeepAlive=yes -o ExitOnForwardFailure=yes -o ServerAliveCountMax=2 -o ServerAliveInterval=10 -N -R PROXYPORT:localhost:22 &>> TUNNEL_LOG\n  echo "Restarting: " `date` >> TUNNEL_LOG\n  sleep 1\ndone' > TUNNEL_SCRIPT_PATH #This creates the above script
echo -e '#!/bin/bash\ncd TUNNEL_SCRIPT_DIRECTORY\n./TUNNEL_SCRIPT_EXECUTABLE &' > /etc/init.d/TUNNEL_SCRIPT_SERVICE_NAME #This creates the init.d daemon script. I have the script set the working path before running the executable so the log file stays in the same directory
chmod u+x TUNNEL_SCRIPT_PATH /etc/init.d/TUNNEL_SCRIPT_SERVICE_NAME #Set all scripts as executable
update-rc.d TUNNEL_SCRIPT_SERVICE_NAME defaults #Set the run levels at which the script runs

I have been having some problems regarding symlinks (symbolic links) for a project that I’ve been working on recently which is requiring work in at least 5 very different operating systems (and about a dozen programming languages). Not many programs support symlinks properly that I have the need to because support for it wasn’t added for NTFS until Windows Vista, and it still has some problems.

It is really great that Windows Vista and Windows 7 now support native symlinks so they can be utilized by programs out of the box. For example, one such instance where I need this a lot is directory relinking in Apache. While Apache’s mod_alias can duplicate the functionality of symlinks for many needs, creating special cases for this one piece of software when distributing a code repository is just not practical, and having proper symlinks natively followed without the program knowing they aren’t the actual file/directory is really the best solution so everything works without special cases.

The way to create NTFS symlinks in Windows Vista+ is through the “mklink” command, which is unfortunately implemented directly in the Window’s command shell, and not a separate executable, so it is not accessible to Cygwin. Further, Cygwin has made a stance to only support reading NTFS symlinks, and not creating them, because they can only be created by administrators, and require specification as to whether the link’s target is a directory or file. Cygwin itself in Windows has had support for symlinks for a long time, but these are not compatible with any program run outside of the Cygwin environment.

Now, my real problem started occurring when trying to use these NTFS symlinks with GIT. While GIT natively supports symlinks, TortoiseGIT doesn’t really support them at all, and throws errors when they are encountered. This is still a big problem that I am going to have to think about :-\. Fortunately, when working with GIT in Cygwin they still work, with caveats. As previously mentioned, only reading the NTFS symlinks in Cygwin work, so when you fetch/pull from a repository and it creates Cygwin style symlinks, Windows still does not read them properly. The following is a script I wrote to change the Cygwin style symlinks into NTFS style symlinks. It can be run from the root folder of the GIT project.

#!/bin/bash
IFS=$'\n' #Spaces do not count as new delimiters

function makewinlink
{
	LINK=$1
	OPTIONS=$2
	TARGET=`find $LINK -maxdepth 0 -printf %l`
	LASTMODTIME=`find $LINK -maxdepth 0 -printf "%t"`
	LINKDIR=`find $LINK -maxdepth 0 -printf %h`
	TARGET=`echo $LINKDIR/$TARGET`
	rm -f $LINK
	cmd /c mklink $OPTIONS "$(cygpath -wa $LINK)" "$(cygpath -wa $TARGET)"
	touch -h -d $LASTMODTIME $LINK
}

#Relink all directories
FILES=`find -type l -print0 | xargs -0 -i find -L {} -type d -maxdepth 0`
for f in $FILES
do
	makewinlink $f /D
done

#Relink all files
FILES=`find -type l -print0 | xargs -0 -i find -L {} -type f -maxdepth 0`
for f in $FILES
do
	makewinlink $f
done

Make sure when committing symlinks in a GIT repository in Windows to use Cygwin with Cygwin style symlinks instead of TortoiseGIT. Also, as previously mentioned, after running this script, TortoiseGIT will show these symlinks as modified :-\. If this is a problem, you can always reverse the process in Cygwin by changing the “cmd /c mklink $OPTIONS” line to a “ln -s” in the above script (note that “target” and “symlink’s name” need to be switched) along with a few other changes.

One thing I always really miss when working in other dynamic languages that aren’t JavaScript is the ability to access known (non dynamic) members of an associative array/object/hash (called a hash from here on out) through just a single dot. This matches C’s syntax of accessing struct members, as opposed to being forced into using array syntax which is harder to read IMO in languages like PHP and Perl. For example...

Zones[Info['Zone']]['DateInfo'][Info['Date']]['UniqueEnters']+=Info['Count'];

Zones[Info.Zone].DateInfo[Info.Date].UniqueEnters+=Info.Count;

I was running into a rather nasty .NET crash today in C# for a rather large project that I have been continuing development for on a handheld device that runs Windows CE6. When I was calling a callback function pointer (called a Delegate in .NET land) from a module, I was getting a TypeLoadException error with no further information. I started out making the incorrect assumption that I was doing something wrong with Delegates, as C# is not exactly my primary language ;-). The symptoms were pointing to the delegate call being the problem because the program was crashing during the delegate call itself, as the code reached the call, and did not make it into the callback function. After doing the normal debugging-thing, I found out the program crashed in the same manner every time the specific callback function was called and before it started executing, even if it was called in a normal fashion from the same class.

After further poking around, I realized that there was one line of code in the function that if included in any function, would cause the program to fail out on calling said function. Basically, resources were somehow missing from the compilation and there were no warnings anywhere telling me this. If I tried to access said resource normally, I was getting an easily traceable MissingManifestResourceException error. However, the weird situation was happening because I had the missing resource being accessed from a static member in another class. So here is some example code that was causing the problem:

public class ClassA
{
	public void PlaySuccess()
	{
		//Execution DOES NOT reach here
		Sound.Play(Sound.Success);
	}
}

public class Sound
{
	public static byte[] Success=MyResource.Success; //This resource is somehow missing from the executable
	public static byte[] Failure=MyResource.Failure;
	public static void Play(byte[] TheSound) { sndPlaySound(TheSound, SND_ASYNC|SND_MEMORY); }
}

ClassA Foo=new ClassA();
//Execution reaches here
Foo.PlaySuccess();

Oh well, at least it wasn’t an array overrun, those are fun to track down :-).

I’ve been really annoyed for a while by the unintuitive IDE tab ordering in Visual Studio 2005+. When you type [shift+]alt+tab, you don’t get the next/previous tab in the list as would be the OBVIOUS way to do it (which probably all other IDEs do this right). No, it switches between tabs in an arbitrary hidden order related to the last acces order of the tabs.

Searching the internet for a solution to this was pretty fruitless, so I tried to tackle the problem myself. I dug through all the possible structures I could find in the Visual Studio IDE macro explorer, and was unfortunately unable to find where the tab order was kept in a window pane (if it is even accessible to the user). I thought I had the solution at one point, but realized it also just switches tabs in the order they were originally opened :-(. This is the VB macro code I came up with to do at least that, which uses “DTE.ActiveWindow.Document.Collection” for the tab-open order.

	 Public Sub TabDirection(ByVal Direction As Integer)
		  'Find the index of the current tab
		  Dim i As Integer
		  Dim Index As Integer
		  Dim Count As Integer
		  Count = DTE.ActiveWindow.Document.Collection.Count
		  For i = 1 To Count
				If DTE.ActiveWindow.Document.Collection.Item(i).ActiveWindow.Equals(DTE.ActiveWindow) Then Index = i
		  Next i

		  'Determine the new index
		  Index = Index + Direction
		  If Index > Count Then Index = 1
		  If Index = 0 Then Index = Count

		  DTE.ActiveWindow.Document.Collection.Item(Index).Activate() 'Activate the next tab in the proper direction
	 End Sub

	 Public Sub TabForward()
		  TabDirection(1)
	 End Sub

	 Public Sub TabBackward()
		  TabDirection(-1)
	 End Sub

Back in May of 2007 one of my friends got me onto Second Life, the first and only MMORPG I’ve touch since my Ragnarok days. While Second Life had a strong pull for me due to its similarities to The MetaVerse in Snow Crash, my favorite book, I was of course more drawn to playing with the Engine and seeing what I could do with it.

I felt no real need to delve into the code or packet level of the client as it was open source, so I stayed mostly on the scripting level side of things in the world. IIRC I did find at least a dozen major security holes, but I unfortunately cannot seem to find logs of my research :-(.

• While an avatar could not pass through solid walls normally, if an object was visible that allowed “sitting” beyond the walls, the user could issue the sit command on that object which transported the avatar past the barriers.
• While there were optional restrictions on areas pertaining to if/where an object could be placed, once an object was placed somewhere, it could be “pushed” to almost any other location no matter the restrictions. When an object was pushed into another area beyond where it was placed, it was still inventoried as being in the originally placed location, but could interact with the world at the location it was actually at. Objects could even pass through solid barriers if the proper push velocities were given. The only way at the time to combat this was to have whole private islands as blocking anonymous objects. This security hole opened up multiple other security holes including:
  • If a user “sat” on the object, they could get to anywhere the object could.
  • These objects could be used to interact with the immediate world around them, including repeating private conversations in a private area.

I had also at the time planned on writing an application that allowed hijacking and reuploading any encountered texture or construct, which was trivial due to the open nature of the system. I never did get around to it for two reasons. First, I got distracted by other projects, and second, because it could have seriously destabilized the Second Life economy, which was built around selling said textures and constructs. I actually liked what Second Life was trying to accomplish and had no wish of making Linden Lab’s life harder or ruining the experiment in open economy.

I was however able to find a few pieces of my research and scripts that I figured I could post here. First, I do not recall what I did to find this, but the entire list of pre-defined “Last Names” was accessible, and IIRC the proprietary last names could be used for character creation if you knew how to access them (not 100% sure if this latter hack was available). Here was the list as of when I acquired it in 2007. I had the list separated into two columns, and I think they were “open” names and “proprietary” names. Each name is followed by its identifier.

The second piece I was able to find was a script I used to alert me via email whenever one of my friends signed on. I have unfortunately not tested this script before posting it as I no longer have Second Life installed or wish to waste the time testing it, but here it is none the less. ^_^;

//Users to watch
key DetectPersons=[ //List of UIDs of users to watch. (Real UIDs redacted)
    "fdf1fbff-f19f-ffff-ffff-ffffffffffff", //Person 1
    "f0fffaff-f61f-ffff-ffff-ffffffffffff" //Person 2
];

//Other Global Variables
integer NumUsers;
integer UsersParsed=0;
list UserNames;
list Status;

default
{
    state_entry()
    {
        NumUsers=llGetListLength(DetectPersons); //Number of users to watch

        //Get User Names
        integer i;
        for(i=0;i<NumUsers;i++)
        {
            llListInsertList(UserNames, [''], i);
            llListInsertList(Status, [0], i);
            llRequestAgentData(llList2Key(DetectPersons, i), DATA_NAME);
        }
    }

    dataserver(key requested, string data)
    {
        //Find User Position
        integer i;
        for(i=0;i<NumUsers;i++)
            if(llList2Key(DetectPersons, i)==requested)
                llListReplaceList(UserNames, [data], i, 1);

        if(++UsersParsed==NumUsers)
            state Running;
    }
}

state Running
{
    state_entry()
    {
        llOwnerSay((string)UserNames);
        llOwnerSay((string)Status);
        llSetTimerEvent(30);
    }

    timer()
    {
        llRequestAgentData(DetectPerson, DATA_ONLINE);
    }

    dataserver(key requested, string data)
    {
        if(data==IsOnline)
            return;
        IsOnline=data;
        if(data=="0")
            return;
        string Message="The user you are watching '"+UserName+"' signed on at "+llGetTimestamp();
        llEmail(EMAIL_ADDRESS, "User Signed on", Message);
        llOwnerSay(Message);
    }
}

Of course all this research was from 2007 and I have no idea what is capable now. I do really hope though that they at least updated the client’s interface because it was incredibly clunky. Also, Second Life has always been a neat experiment, and I hope it still is and continues to keep doing well :-).

While UTF-8 has almost universally been accepted as the de-facto standard for Unicode character encoding in most non-Windows systems (mmmmmm Plan 9 ^_^), the BOM (Byte Order Marker) still has large adoption problems. While I have been allowing my text editors to add the UTF8 BOM to the beginning of all my text files for years, I have finally decided to rescind this practice for compatibility reasons.

While the UTF8 BOM is useful so that editors know for sure what the character encoding of a file is, and don’t have to guess, they are not really supported, for their reasons, in Unixland. Having to code solutions around this was becoming cumbersome. Programs like vi and pico/nano seem to ignore a file’s character encoding anyways and adopt the character encoding of the current terminal session.

The main culprit in which I was running into this problem a lot with is PHP. The funny thing about it too was that I had a solution for it working properly in Linux, but not Windows :-).

Web browsers do not expect to receive the BOM marker at the beginning of files, and if they encounter it, may have serious problems. For example, in a certain browser (*cough*IE*cough*) having a BOM on a file will cause the browser to not properly read the DOCTYPE, which can cause all sorts of nasty compatibility issues.

Something in my LAMP setup on my cPanel systems was removing the initial BOM at the beginning of outputted PHP contents, but through some preliminary research I could not find out why this was not occurring in Windows. However, both systems were receiving multiple BOMs at the beginning of the output due to PHP’s include/require functions not stripping the BOM from those included files. My solution to this was a simple overload of these include functions as follows (only required when called from any directly opened [non-included] PHP file):

<?
/*Safe include/require functions that make sure UTF8 BOM is not output
Use like: eval(safe_INCLUDETYPE($INCLUDE_FILE_NAME));
where INCLUDETYPE is one of the following: include, require, include_once, require_once
An eval statement is used to maintain current scope
*/

//The different include type functions
function safe_include($FileName)	{ return real_safe_include($FileName, 'include'); }
function safe_require($FileName)	{ return real_safe_include($FileName, 'require'); }
function safe_include_once($FileName)	{ return real_safe_include($FileName, 'include_once'); }
function safe_require_once($FileName)	{ return real_safe_include($FileName, 'require_once'); }

//Start the processing and return the eval statement
function real_safe_include($FileName, $IncludeType)
{
	ob_start();
	return "$IncludeType('".strtr($FileName, Array("\\"=>"\\\\", "'", "\\'"))."'); safe_output_handler();";
}

//Do the actual processing and return the include data
function safe_output_handler()
{
	$Output=ob_get_clean();
	while(substr($Output, 0, 3)=='?') //Remove all instances of UTF8 BOM at the beginning of the output
		$Output=substr($Output, 3);
	print $Output;
}
?>

I would have like to have used PHP’s output_handler ini setting to catch even the root file’s BOM and not require include function overloads, but, as php.net puts it “Only built-in functions can be used with this directive. For user defined functions, use ob_start().”.

As a bonus, the following bash command can be used to find all PHP files in the current directory tree with a UTF8 BOM:

grep -rlP "^\xef\xbb\xbf" . | grep -iP "\.php\$"

Seeing as there are a number of applications on the market that have both a “Free” and “Full” version, you’d think this would be an easy thing to accomplish. Unfortunately, the marketplace uses an application’s package name as its unique identifier, so both versions have to have a different package name, which is again, a bit of a nuisance.

One method of remedying this is just having a recursive string replace through all the files [...]

I spent a bit of time coming up with a solution for this a few months ago for my latest project, [TEMPORARILY DOWN**]. This solution uses a shared package with a shared code base and resources that the different project versions pull from.

**The project that is a great example of how this process works should be uploaded very soon. At that time this message will disappear and appropriate links will be added. You’ll know this has happened when I upload my next project.

• First some definitions that will be used below*:
  • ProjectName: The base name of the project (e.g. “MyAndroidProject”)
  • VersionName: The name of separate versions (e.g. “Free” and “Full”)
  • SharedName: The name of the shared section of the code (e.g. “Shared”)
  • BasePackageName: The base name of the package group (e.g. “com.example.MyAndroidProject”)
  • BasePackageDirectory: The base path of the package group (e.g. “com/example/MyAndroidProject”)
  *Please note these definitions are used in code sections below.
• Create the directory structure:
  • A base directory of ProjectName (e.g. “~/MyAndroidProject”)
  • A subdirectory under the base directory named SharedName for the shared files (e.g. “~/MyAndroidProject/Shared”). It will hold any shared files in appropriate Android standard directories (e.g. “res”, “src”).
  • Subdirectories under the base directory named VersionName for each version’s project (e.g. “~/MyAndroidProject/Free”). Each of these will hold its own complete project including the AndroidManifest.
• Creating the shared resources: There’s nothing special about shared resources (probably in “SharedName/res”), except I suggest noting at the top of the files that they are shared, for reference sake.
• Creating the shared code:
  • Shared code goes in “SharedName/src/BasePackageDirectory/SharedName” (e.g. “~/MyAndroidProject/Shared/src/com/example/MyAndroidProject/Shared”).
  • As noted for shared resources, you might want to note at the top of the files that they are shared.
  • Set the package name for shared code to “BasePackageName.SharedName” (e.g. “package com.example.MyAndroidProject.Shared;”).
  • Shared code should never directly refer back to a version’s package (non shared) code except through reflection.
  • Resource IDs are still accessible in this shared package through the “R” class, but when accessed the function or class that does the accessing needs to be proceeded with “@SuppressWarnings({ "static-access" })”. The “R” variable also has a “Version” member that can be used to alter shared code flow depending on the version being used. This will be explained more in detail later.
  • *BONUS*
    If shared code needs to access information from a static member in a non-shared class, reflection can be used, for example:

    Class.forName("BasePackageName."+R.Version.name()+".NonSharedClassName").getDeclaredField("StaticMemberName").get(null)

    A static function can be called in a similar manner through reflection:

    Class.forName("BasePackageName."+R.Version.name()+".NonSharedClassName").getMethod("StaticMethodName", new Class[0]).invoke(null);

• *BONUS* Global Shared Class: I also suggest having a shared class that holds global variables that allows easy interaction from the shared to non shared classes, and holds static information that both use, with members including:
  • A reference to a Context (or Activity) from the main program
  • The BasePackageName (needed for reflection, and other stuff like preference storage)
  • Other useful interfaces like logging
• Creating the non-shared code:
  • Create a separate project for each version in their corresponding subdirectories listed in the third step of the Create the directory structure" section above.
  • Make the package name for a version as “BasePackageName.VersionName”.
  • When referencing shared classes form an android manifest, make sure to use their full package name, for example, a shared activity would look like “<activity android:name="BasePackageName.SharedName.ActivityName">”
  • Import the shared package into all non-shared class files “import BasePackageName.SharedName.*;”
• Linking the shared code into each project:
  • The shared code now needs to get integrated into each project. To do this, all the shared files need to be symbolically (or hard) linked back into their corresponding directories for each version.
  • First, make sure each project directory also contains the same subdirectories as those found in the shared directory.
  • The script I have written for this, which needs to be put in the base directory, is as follows: [EDIT ON 2011-01-03 @ 6:28am] See here for a better copy of the script. [/EDIT]

    #!/bin/bash
    
    #Run this file to install links to shared files into all branches
    
    LN="ln" #Use hard links, which work well in Windows and require less logic to calculate linking
    
    #if [ `uname || grep -vi 'cygwin'` ]; then #If not windows (NOT YET SUPPORTED)
    #	LN="ln -s" #Use symbolic links, which take some additional logic that is not yet programmed
    #fi
    
    LocalBranches=`find -maxdepth 1 -type d | grep -iPv "^\.(/SharedName|)$"` #Find version names, ignoring "." ".." and the shared directory
    
    #Propagate shared files into different versions
    cd Shared
    for i in $LocalBranches; do
    	find -type f -print0 | xargs -0 -i rm -f ../$i/{} #Clear out old links from version just in case the link has been undone
    	if [ "$1" != "clean" ]; then
    		find -type f -print0 | xargs -0 -i $LN {} ../$i/{} #Link shared files into directories
    	fi
    done
    				

• Tying the resources together:
  • The resources IDs in the “R” class might need to be accessible by the shared classes. To do this, an “R” class needs to be added into the shared namespace that duplicates the information. Unfortunately, the Android generated resources ID class “R” marks everything as “final” so class duplication requires a bit more work than would be preferable.
  • Each different version needs its own “R” class, put into the directory “~/MyAndroidProject/VersionName/src/BasePackageDirectory/SharedName” that the shared code reads from.
  • This class will also contain version information so the shared classes can know which version it is currently interacting with.
  • The code is as follows:

    package BasePackageName.SharedName;
    
    import BasePackageName.VersionName.R.*;
    
    public final class R //Mimic the resources from the (non-shared) parent project
    {
    	//There may be more resource ID groups than these that need to be shared
    	final static attr attr=null;
    	final static drawable drawable=null;
    	final static id id=null;
    	final static layout layout=null;
    	final static string string=null;
    	final static xml xml=null;
    	final static array array=null;
    	
    	public enum Versions { Version1, Version2 }; //List versions here
    	
    	public final static Versions Version=Versions.CurrentVersion;
    }
    			

  • Whenever this “shared” “R” class is accessed, the function or class that does the accessing needs to be proceeded with “@SuppressWarnings({ "static-access" })”. This is due to the hack required to reproduce the “final” class information from the original “R” class into a shared class.
• Working with shared projects and code in Eclipse:
  • When modifying shared code in Eclipse for any of the different versions, other version’s projects need to be refreshed to pick up the new code. I tried many different methods and processes to fix this different package name problem but this described method is still much quicker and easier than any of the others.

Having recently finished my first Android project (and hopefully not last), I decided to supply some notes I took about the process.

While I am going to try and keep impressions to a minimum on the rest of this post, and keep it to tangible notes, I must first comment that trying to find out things for the Android platform was often like pulling teeth. While its typical Java reference online documentation is all there with all the classes and cross-linking, that is about all it is, very dry and virtually useless beyond a base reference. The comments on variable parameters (and many other sections) in the reference are often coarse and not descriptive at all, for example, one parameter named mask has the basic description as “this is a mask”. Some functions don’t even have descriptions at all.

Perhaps I am getting too complacent as a programmer and getting used to excellent documentation like for Python or GTK (I’ve even grown to love Microsoft documentation after having used it for long enough!). After all, most required information is just a Google away, and being a programmer is often just about finding the proper magical incantations to hook into a certain library. Unfortunately, however, even web searches were often yielding less than fruitful results when dealing with Android, as the platform is relatively new.

• Some useful tasks and some problems:
  • Using the virtual (soft) keyboard without a TextView:
    • Showing the virtual keyboard:

      ((InputMethodManager)getSystemService(INPUT_METHOD_SERVICE)).toggleSoftInput(InputMethodManager.SHOW_FORCED, InputMethodManager.HIDE_IMPLICIT_ONLY);

    • Hiding the virtual keyboard:

      ((InputMethodManager)getSystemService(INPUT_METHOD_SERVICE)).hideSoftInputFromWindow(getWindow().getDecorView().getApplicationWindowToken(), 0);

      Note: “getWindow().getDecorView()” can also be replaced by a View on your screen
    • Getting the keyboard input: Add the following function to the Activity that opened the keyboard:

      @Override public boolean onKeyDown(int keyCode, KeyEvent msg)

      Note: This will not work if you’re not using default keyboard input (like if it’s set to enter Japanese or Chinese characters).
  • Determining the physical dimensions of the screen:

    This should be a trivial task using the DisplayMetrics (getWindowManager().getDefaultDisplay()) interface to get dpis and multiply by the screen dimensions getWindowManager().getDefaultDisplay().getWidth() (and .getHeight). However, it doesn’t always work as it should.

    The best method to get the DPI would be to use “DisplayMetrics.xdpi” and “DisplayMetrics.ydpi”, but unfortunately, these are misreported by at least the Motorola Droid. I’ve found “DisplayMetrics.density”*160 to be pretty accurate, but if true accuracy is needed, a calibration screen might be required.

  • Inform user of touch events: Many Android widgets (Views) change their visual state (highlight) when the user presses down on them to let the user know something is going to happen if the user lifts their finger while still on the widget. Unfortunately, there seems to be no text widget or layout view that does this automatic highlighting by itself (ListViews do in groups). The following is some example code to produce this effect.

    import android.view.View.OnTouchListener;
    
    public class CLASSNAME extends Activity
    {
    	@Override public void onCreate(Bundle savedInstanceState)
    	{
    		View HighlightView=findViewById(R.id.THE_VIEWS_ID);
    		HighlightView.setOnTouchListener(HighlightState);
    	}	
    	
    	private OnTouchListener HighlightState = new OnTouchListener() { public boolean onTouch(View v, MotionEvent event)
    	{
    		if(event.getAction()==MotionEvent.ACTION_DOWN)
    			v.setBackgroundColor(0xFF0000FF); //Set background color to blue
    		else if(event.getAction()==MotionEvent.ACTION_CANCEL || event.getAction()==MotionEvent.ACTION_UP)
    			v.setBackgroundResource(0); //No background color
    			
    		return false;
    	} };
    }

  • Retrieving the names and IDs of all resources in a resource group:

    import java.lang.reflect.Field;
    
    Field[] FieldList=R.drawable.class.getDeclaredFields();
    String[] Names=new String[FieldList.length];
    int[] IDs=new int[FieldList.length];
    for(int i=0;i<FieldList.length;i++)
    	IDs[i]=getResources().getIdentifier(Names[i]=FieldList[i].getName(), "drawable", getClass().getPackage().getName());

  • Setting a color matrix on an image: If you have 2 ImageViews that display the same resource image, and either has a color matrix set on it, the will both share one of the color matrices. If this occurs, copy the image the resource, or use a separate image resource. For kicks, here is an example of setting an inverse color matrix on an image.

    ((ImageView)findViewById(R.id.IMAGE_ID)).setColorFilter(new ColorMatrixColorFilter(new float[] {-1,0,0,0,255, 0,-1,0,0,255, 0,0,-1,0,255, 0,0,0,1,0}));

  • Setting to full screen:

    requestWindowFeature(Window.FEATURE_NO_TITLE); //This must be called before "setContentView", and hides the title bar
    getWindow().setFlags(FULLSCREEN ? WindowManager.LayoutParams.FLAG_FULLSCREEN : 0, WindowManager.LayoutParams.FLAG_FULLSCREEN); //Turns on/off the status bar

  • Starting another local activity: Instead of using Intent(String action) for Context.StartActivity, as the Reference explains, it is much easier to use Intent(Context packageContext, Class<?> cls) like the following: (called from inside an Activity)

    startActivity(new Intent(this, OTHER_ACTIVITY_NAME.class);

  • Creating a timed event that updates the UI: A function running through java.util.Timer cannot interact with the GUI. One solution to make a timer is with the android.os.Handler interface.

    import android.os.Handler;
    
    public class ExampleActivity extends Activity
    {
    	final int InitialDelay, RepeatDelay; 
    	Handler TimedHandler=new Handler();
    	
    	public void ExampleFunction()
    	{
    		TimedHandler.postDelayed(new Runnable() { public void run() {
    			//Do GUI stuff...
    			TimedHandler.postDelayed(this, RepeatDelay);
    		} }, InitialDelay);
    	}
    }

    
    Another solution is to post to a Handler from the Timer function.
• When dealing with putting on the market place:
  • Getting an account to put applications on the Android Market cost $25.
  • Screenshots shown on the Android Market description page are somewhat buggy, and seemingly randomly either stretch properly or crop. Viewing the full sized screenshots does seem to work properly.
  • Seeing as there are a number of applications on the market that have both a “Free” and “Full” version, you’d think this would be an easy thing to accomplish. Unfortunately, the marketplace uses an application’s package name as its unique identifier, so both versions have to have a different package name, which is again, a bit of a nuisance.

    One method of remedying this is just having a recursive string replace through all the files to change the package names. However, if using eclipse, so you don’t have to reopen it, it’s quicker to update the string first in the manifest, and then renaming the package under the “src” folder by pressing F2 (rename) on it when it is selected.

    Also, unfortunately, if you do this, when a person upgrades from the lite to the full version, preferences are not automatically transferred :-\.

  • The publisher’s market place page is very sparse and leaves a lot to be desired. It also seems to update only once every 24 hours or so (not sure of exact times).
  • If an application is put up, it WILL get downloads immediately. For example, I put up an application with a description of “This is a test, do not download this” for doing security tests that I took down within like 10 minutes. It already had 2 comments/ratings on it within that time ~.~; .
  • Google Checkout: Fees. When a copy of your application is purchased, the user has 24 hours to return it. The money is not deposited into your bank account until after this time (when it’s not a weekend). If you want to give your application to someone for free, they need to purchase it through the market, and then you can cancel the purchase transaction before the 24 hours are up. Unfortunately, this has to be done every time they want to update the application. It also seems you cannot buy your own applications, as the purchase server throws an error.
• Application Protection:

  You can download any Android application by default from your phone to your computer, modify them, and reinstall them back to any phone. An example use for this would be to crack a shareware application where just a single byte probably needs to be changed to make it a full version.

  The applications themselves are in an .apk file (which is just a .zip file), and the source code (classes) are encoded as a “Dalvik Executable” file within it (classes.dex), which as I understand it, is optimized Java bytecode. So, AFAIK right now, there is no way to decompile the .dex file back to the original source, like you can with normal Java. However, the Android emulator, part of the Android SDK, includes a tool called dexdump, which allows you to decompile it to bytecode.

  Once you have the bytecode, you can use that as reference to modify the compiled .dex file however you want, which is pretty much synonymous with assembly editing. Once that is done, the signature and checksum of the .dex file must be recalculated (Java source by Timothy Strazzere), and then the apk file must be resigned, and then it’s good to go!

  The marketplace also has an option to turn on Copy Protection. When this is turned on for an application, the user cannot backup or access the applications package file. I would assume however with a rooted phone you could still grab it from “/data/app-private”, and the rest of the process should be the same. I have not tested this as rooting Android 2.1 is much more of a pain in the butt, ATM, than I want to deal with.

I had the need to pass a program’s [standard] output to a web browser in real time. The best solution for this is to use a combination of programs made in different languages. The following are all of these individual components to accomplish this task.

Please note the C components are only compatible with gcc and bash (cygwin required for Windows), as MSVC and Windows command prompt are missing vital functionality for this to work.

The first component is a server made in C that receives stdin (as a pipe, or typed by the user after line breaks) and sends that data out to a connected client (buffering the output until the client connects).

PassThruServer source, PassThruServer compiled Windows executable.

• This compiles as C99 under gcc:

  gcc PassThruServer.c -o PassThruServer

• Define “WINDOWS” when compiling in Windows (pass “-DWINDOWS”)

#include <stdio.h>
#include <malloc.h>
#include <fcntl.h>
#include <sys/types.h> 
#include <sys/socket.h>
#include <netinet/in.h>
#include <signal.h>

//The server socket and options
int ServerSocket=0;
const int PortNumber=1234; //The port number to listen in on

//If an error occurs, exit cleanly
int error(char *msg)
{
	//Close the socket if it is still open
	if(ServerSocket)
		close(ServerSocket);
	ServerSocket=0;

	//Output the error message, and return the exit status
	fprintf(stderr, "%s\n", msg);
	return 1;
}

//Termination signals
void TerminationSignal(int sig)
{
	error("SIGNAL causing end of process");
	_exit(sig);
}

int main(int argc, char *argv[])
{
	//Listen for termination signals
	signal(SIGINT, TerminationSignal);
	signal(SIGTERM, TerminationSignal);
	signal(SIGHUP, SIG_IGN); //We want the server to continue running if the environment is closed, so SIGHUP is ignored -- This doesn't work in Windows
	
	//Create the server
	struct sockaddr_in ServerAddr={AF_INET, htons(PortNumber), INADDR_ANY, 0}; //Address/port to listen on
	if((ServerSocket=socket(AF_INET, SOCK_STREAM, 0))<0) //Attempt to create the socket
		return error("ERROR on 'socket' call");
	if(bind(ServerSocket, (struct sockaddr*)&ServerAddr, sizeof(ServerAddr))<0) //Bind the socket to the requested address/port
		return error("ERROR on 'bind' call");
	if(listen(ServerSocket,5)<0) //Attempt to listen on the requested address/port
		return error("ERROR on 'listen' call");

	//Accept a connection from a client
	struct sockaddr_in ClientAddr;
	int ClientAddrLen=sizeof(ClientAddr);
	int ClientSocket=accept(ServerSocket, (struct sockaddr*)&ClientAddr, &ClientAddrLen);
	if(ClientSocket<0) 
		return error("ERROR on 'accept' call");

	//Prepare to receive info from STDIN
		//Create the buffer
		const int BufferSize=1024*10;
		char *Buffer=malloc(BufferSize); //Allocate a 10k buffer
		//STDIN only needs to be set to binary mode in windows
		const int STDINno=fileno(stdin);
		#ifdef WINDOWS
			_setmode(STDINno, _O_BINARY);
		#endif
		//Prepare for blocked listening (select function)
		fcntl(STDINno, F_SETFL, fcntl(STDINno, F_GETFL, 0)|O_NONBLOCK); //Set STDIN as blocking
		fd_set WaitForSTDIN;
		FD_ZERO(&WaitForSTDIN);
		FD_SET(STDINno, &WaitForSTDIN);

	//Receive information from STDIN, and pass directly to the client
	int RetVal=0;
	while(1)
	{
		//Get the next block of data from STDIN
		select(STDINno+1, &WaitForSTDIN, NULL, NULL, NULL); //Wait for data
		size_t AmountRead=fread(Buffer, 1, BufferSize, stdin); //Read the data
		if(feof(stdin) || AmountRead==0) //If input is closed, process is complete
			break;
		
		//Send the data to the client
		if(write(ClientSocket,Buffer,AmountRead)<0) //If error in network connection occurred
		{
			RetVal=error("ERROR on 'write' call");
			break;
		}
	}
	
	//Cleanup
	if(ServerSocket)
		close(ServerSocket);
	free(Buffer);
	
	return RetVal;
}

The next component is a Flash applet as the client to receive data. Flash is needed as it can keep a socket open for realtime communication. The applet receives the data and then passes it through to JavaScript for final processing.

Compiled Flash Client Applet

import flash.external.ExternalInterface;
import flash.events.Event;
ExternalInterface.addCallback("OpenSocket", OpenSocket);

function OpenSocket(IP:String, Port:Number):void
{
	SendInfoToJS("Trying to connect");
	var TheSocket:Socket = new Socket();
	TheSocket.addEventListener(Event.CONNECT, function(Success) { SendInfoToJS(Success ? "Connected!" : "Could not connect"); });
	TheSocket.addEventListener(Event.CLOSE, function() { SendInfoToJS("Connection Closed"); });
	TheSocket.addEventListener(IOErrorEvent.IO_ERROR, function() {SendInfoToJS("Could not connect");});
	TheSocket.addEventListener(ProgressEvent.SOCKET_DATA, function(event:ProgressEvent):void { ExternalInterface.call("GetPacket", TheSocket.readUTFBytes(TheSocket.bytesAvailable)); });
	TheSocket.connect(IP, Port);
}
function SendInfoToJS(str:String) { ExternalInterface.call("GetInfoFromFlash", str); }
stop();

var NewSock=new XMLSocket();
NewSock.onData=function(msg) { GetPacket(msg); }
NewSock.onConnect=function(Success) { SendInfoToJS(Success ? "Connected!" : "Could not connect"); }
SendInfoToJS(NewSock.connect(IP, Port) ? "Trying to Connect" : "Could not start connecting");

JavaScript can then receive (and send) information from (and to) the Flash applet through the following functions.

• FLASH.OpenSocket(String IP, Number Port): Call this from JavaScript to open a connection to a server. Note the IP MIGHT have to be the domain the script is running on for security errors to not be thrown.
• JAVASCRIPT.GetInfoFromFlash(String): This is called from Flash whenever connection information is updated. I have it giving arbitrary strings ATM.
• JAVASCRIPT.GetPacket(String): This is called from Flash whenever data is received through the connection.

This example allows the user to input the IP to connect to that is streaming the output. Connection information is shown in the “ConnectionInfo” DOM object. Received data packets are appended to the document in separate DOM objects.

JavaScript+HTML Source

var isIE=navigator.appName.indexOf("Microsoft")!=-1;
function getFlashMovie(movieName) { return (isIE ? window[movieName] : document[movieName]);  }
function $(s) { return document.getElementById(s); }

function Connect()
{
	getFlashMovie("client").OpenSocket($('IP').value, 1234);
}

function GetInfoFromFlash(Str)
{
	$('ConnectionInfo').firstChild.data=Str;
}

function GetPacket(Str)
{
	var NewDiv=document.createElement('DIV');
	NewDiv.appendChild(document.createTextNode(Str));
	$('Info').appendChild(NewDiv);
}

Next is an example application that outputs to stdout. It is important that it flushes stdout after every output or the communication may not be real time.

inc source, inc compiled Windows executable.

inc counts from 0 to one less than a number (parameter #1 [default=50]) after a certain millisecond interval (parameter #2 [default=500]).

./inc 10 #Counts from 0-9 every half a second

#include <stdio.h>
#include <stdlib.h>

int main(int argc, char *argv[])
{
	int NumLoops=(argc>1 ? atoi(argv[1]) : 50); //Number of loops to run from passed argument 1. Default is 50 if not specified.
	int LoopWait=(argc>2 ? atoi(argv[2]) : 500); //Number of milliseconds to wait in between each loop from passed argument 2. Default is 500ms if not specified.
	LoopWait*=1000; //Convert to microseconds for usleep

	//Output an incremented number every half a second
	int i=0;
	while(i<NumLoops)
	{
		printf("%u\n", i++);
		fflush(stdout); //Force stdout flush
		usleep(LoopWait); //Wait for half a second
	};
	
	return 0;
}

This final component is needed so the Flash applet can connect to a server. Unfortunately, new versions of Flash (at least version 10, might have been before that though) started requiring policies for socket connections >:-(. I don’t think this is a problem if you compile your applet to target an older version of Flash with the ActionScript v1.0 code.

This Perl script creates a server on port 843 to respond to Flash policy requests, telling any Flash applet from any domain to allow connections to go through to any port on the computer (IP). It requires Perl, and root privileges on Linux to bind to a port <1024 (su to root or run with sudo).

Flash Socket Policy Server (Rename extension to .pl)

#!/usr/bin/perl
use warnings;
use strict;

#Listen for kill signals
$SIG{'QUIT'}=$SIG{'INT'}=$SIG{__DIE__} = sub
{
	close Server;
	print "Socket Policy Server Ended: $_[0]\n";
	exit;
};

#Start the server:
use Socket;
use IO::Handle;
my $FlashPolicyPort=843;
socket(Server, PF_INET, SOCK_STREAM, getprotobyname('tcp')) or die "'socket' call: $!"; #Open the socket
setsockopt(Server, SOL_SOCKET, SO_REUSEADDR, 1) or die "'setsockopt' call: $!"; #Allow reusing of port/address if in TIME_WAIT state
bind(Server, sockaddr_in($FlashPolicyPort,INADDR_ANY)) or die "'bind' call: $!"; #Listen on port $FlashPolicyPort for connections from any INET adapter
listen(Server,SOMAXCONN) or die "'listen' call: $!"; #Start listening for connections
Server->autoflush(1); #Do not buffer output

#Infinite loop that accepts connections
$/ = "\0"; #Reset terminator from new line to null char
while(my $paddr=accept(Client,Server))
{
	Client->autoflush(1); #Do not buffer IO
	if(<Client> =~ /.*policy\-file.*/i) { #If client requests policy file...
		print Client '<cross-domain-policy><allow-access-from domain="*" to-ports="*" /></cross-domain-policy>'.$/; #Output policy info: Allow any flash applets from any domain to connect
	}
	close Client; #Close the client
}

1. Start the servers
   • In your [bash] command shell, execute the following

     Server/FlashSocketPolicy.pl & #Run the Flash Policy Server as a daemon. Don't forget sudo in Linux
     ./inc | ./PassThruServer #Pipe inc out to the PassThruServer

   • Note that this will immediately start the PassThruServer receiving information from “inc”, so if you don’t get the client up in time, it may already be done counting and send you all the info at once (25 seconds).
   • The PassThruServer will not end until one of the following conditions has been met:
     • The client has connected and the piped process is completed
     • The client has connected and disconnected and the disconnect has been detected (when a packet send failed)
     • It is manually killed through a signal
   • The Flash Policy Server daemon should probably just be left on indefinitely in the background (it only needs to be run once).
2. To run the client, open client.html through a web server [i.e. Apache’s httpd] in your web browser. Don’t open the local file straight through your file system, it needs to be run through a web server for Flash to work correctly.
3. Click “connect” (assuming you are running the PassThruServer already on localhost [the same computer]). You can click “connect” again every time a new PassThruServer is ran.

A client of mine wanted their website to have an applet that played streaming music from a SHOUTcast server. The easy solution would have been to just embed a Windows Media Player applet into the page, but that would only work for IE.

I thoroughly searched the web and was unable to find a Flash applet (or other solution) that already did this (and actually worked). Most of the information I was finding was people having problems getting this kind of thing working in Flash with no answer provided. After giving up on finding a resolution online, I decided to load up Flash and see what I could find from some tinkering.

Quite frankly, I’m shocked people were having so many problems with this. I started an ActionScript 2.0 project and put in the following code, and it worked right away in Flash CS3 (v9.0) with no problem:

var URL="http://example.shoutcast.castledragmire.com:1234/" //The URL to the SHOUTcast server
var MySound:Sound=new Sound(this);
MySound.loadSound(URL,true);

Unfortunately, once I exported the Flash applet and loaded it up in my browsers, it was no longer working. After a few minutes of poking around, I had a hunch that the SHOUTcast host might be sending different data depending on the [Browser’s] User Agent. I changed Firefox’s User Agent to “Flash” through a Firefox add-on (User Agent Switcher), and it worked :-D.

Once again, unfortunately, this was not a viable solution because I couldn’t have every user who visited the client’s web page change their browser User Agent string :-). The quickest solution at this point to the problem was to just create a passthrough script that grabbed the live stream on their server and passed it to the client. The following is the PHP script I used for this:

$streamname='example.shoutcast.castledragmire.com';
$port      ='1234';
$path      ='/';

header('Content-type: audio/mpeg');
$sock=fsockopen($streamname,$port);
fputs($sock, "GET $path HTTP/1.0\r\n");
fputs($sock, "Host: $streamname\r\n");
fputs($sock, "User-Agent: WinampMPEG/2.8\r\n");
fputs($sock, "Accept: */*\r\n");
fputs($sock, "Connection: close\r\n\r\n");
fpassthru($sock);
fclose($sock);

1. Setting the Flash Applet’s URL variable to the PHP file
2. Turning off PHP output buffering for the file. This can only be done through Apache or the php.ini depending on the server setup. This is very important, as if it’s on, the data will never get sent to the user.

The only problem with this method is that it taxes the server that is passing the data through, especially since it uses PHP... This kind of thing could very easily be done in C though (as a matter of fact, I will be writing a post on something very close to that very soon).

To continue the subject in my last post, these next cross-window bugs also derive from objects not being recognized properly when being passed between windows in JavaScript.

I needed the ability to dynamically run functions in the secondary window form the primary window where the parameters are taken from an array. Since a “function” from a secondary window is not seen as a function object from the primary window in IE, the apply member was not working.

I have included a fix for this below in the “RunFunctionInRemoteWindow” function, which is just a wrapper function in the second window that calls the apply function. This function manually copies the array through a for loop, instead of using slice, because in IE7 (but not IE8), the passed arrays were not seen as valid JSObjects, so the slice method (which is a standard method used for copying arrays by value) was not working.

<html><body>
<input type=button onclick="RunTest();" value='Click me when the second window has opened to run the test'>
<script type="text/javascript">

//Spawn the second window
var NewWindow=window.open('RemoteWindow.html');

//Run the test
function RunTest()
{
	LocalAlertString('This is an alert generated from the local window');
	NewWindow.RemoteAlertString('This is an alert generated from the remote window');
	alert('The local window alert function is of type function: '+(LocalAlertString instanceof Function));
	alert('The remote window alert function is of type function: '+(NewWindow.RemoteAlertString instanceof Function));
	LocalAlertString.apply(window, ['This is an alert generated from the local window through the APPLY member']);

	try {
	NewWindow.RemoteAlertString.apply(NewWindow.window, ['This is an alert generated from the remote window through the APPLY member. This will not work in IE because the remote window\'s function is not actually a function.']);
	} catch(e) { alert('The REMOTE APPLY failed: '+e.message); }

	NewWindow.RunFunctionInRemoteWindow('RemoteAlertString', ['This is an alert generated from the remote window through the FIXED APPLY function.']);
}

//Generate an alert in the local window
function LocalAlertString(TheString)
{
	alert('Local String: '+TheString);
}

</script></body></html>

<html><body><script type="text/javascript">
//Generate an alert in the remote window
function RemoteAlertString(TheString)
{
	alert('Remote String: '+TheString);
}

//Call functions in this window remotely through the "apply" member
function RunFunctionInRemoteWindow(FunctionName, Parameters)
{
	//Manually copy the passed Parameters since "Parameters" may not be a valid JSObject anymore (this could be detected and array.slice used if it is still valid)
	var ParametersCopy=[];
	for(var i=0;i<Parameters.length;i++)
		ParametersCopy[i]=Parameters[i];
	
	window[FunctionName].apply(window, ParametersCopy);
}
</script></body></html>

I was doing some research around April of 2009 on JavaScript interaction between web browser windows. I was doing this because web browsers are starting to split off each tab/window into separate processes/threads (Firefox is lagging in this), which can lead to some useful new implementations in the browser world, including multithreading. I wanted to explore the interaction between these windows to make sure there were no caveats that might creep up if I decided to take advantage of this.

The first one I found was that each browser window has its own instance of all of the base object classes, so prototypes do not carry over, and instanceof will not work as expected.

For example, if in WindowOne, you add a prototype to the Array class called IsArray, it is only accessible by arrays created in WindowOne. If you pass an array created in WindowOne into a second window, the prototype is still available on that one array (IIRC this was not true of some of the browsers at the time, but I tested again today, and it worked for IE8, Firefox3, and Google Chrome). Also, since the base object class in Window1 and other windows are not the same, an object created in Window1 and passed to another window will return false in a instanceof Object operation in that other window.

<html><body>
<input type=button onclick="RunTest();" value='Click me when the second window has opened to run the test'>
<script type="text/javascript">
Array.prototype.IsArray=true;
var NewWindow=window.open('RemoteWindow.html'); //Spawn the second window
function RunTest() { NewWindow.RunTest({}, [], new ExampleObject()); }; //Send the test data to remote window
function ExampleObject() { } //An example class
</script></body></html>

<html><body><script type="text/javascript">
function RunTest(AnObject, AnArray, AnExampleObject)
{
   var MyTests=[
      'AnObject instanceof Object',
      'AnObject.IsArray',                               //Object.prototype does not have this (Array.prototype does)
      'AnArray instanceof Object',
      'AnArray instanceof Array',
      'AnArray.IsArray',                                //This was added to the Array.prototype in the parent window
      'AnArray instanceof opener.Array',                //This example does not work in IE7 because opener.x cannot be properly accessed
      'AnExampleObject instanceof opener.ExampleObject',//This example does not work in IE7 because opener.x cannot be properly accessed
      'AnExampleObject instanceof ExampleObject'        //This test should error because "ExampleObject" does not exist in this window
   ];
   
   for(var i=0;i<MyTests.length;i++) //This runs each test like the following: alert("TEST: "+(TEST));
      try {
         eval('alert("'+MyTests[i]+': "+('+MyTests[i]+'));');
      } catch(e) {
         alert('Error on test "'+MyTests[i]+'": '+(e.hasOwnProperty('message') ? e.message : e.toString()));
      }
}
</script></body></html>

Since I just released my AJAX Library, I thought I’d post a useful script that uses it. The function CrossDomainGetURL below uses the AJAX Library to make requests across domains in Firefox. It takes one more parameter (not in order) than the AJAX Library's GetURL function, which is an array of domains to pull cookies from for the AJAX request.

function GetCookiesFromURL(Domains) //Return all the cookies for Domains specified in the Domains array
{
	var cookieManager = Components.classes["@mozilla.org/cookiemanager;1"].getService(Components.interfaces.nsICookieManager); //Requires privileges, which is granted in CrossDomainGetURL
	var iter=cookieManager.enumerator, CookieList=[], cookie; //The object used to find all cookies, the final list of cookies, and a temporary object
	while(iter.hasMoreElements()) //Loop through all cookies
		if(((cookie=iter.getNext()) instanceof Components.interfaces.nsICookie) && Domains.indexOf(cookie.host)!=-1) //If a cookie whose host matches one of our domains
			CookieList.push(cookie.name+'='+cookie.value); //Add it to our final list
	return CookieList.join("; "); //Return the cookie list for the specified domains
}

function CrossDomainGetURL(URL, Data, CookieDomains, ExtraOptions) //See AJAX Library GetURL function. CookieDomains is an array specifying what domains cookies are pulled from for the AJAX call. 
{
	//Access universal privileges in Firefox (Required to get cookies for other domains, and to use AJAX with other domains). This functionality is lost as soon as this function loses scope.
	try { netscape.security.PrivilegeManager.enablePrivilege("UniversalXPConnect"); }
	catch(e) { return alert('Cannot access browser privileges'); }

	if(CookieDomains instanceof Array) //If an array of domains is passed to get cookies from...
	{	
		ExtraOptions=((ExtraOptions instanceof Object) ? ExtraOptions : {}); //Make sure extra options is an object
		ExtraOptions.AdditionalHeaders=((ExtraOptions.AdditionalHeaders instanceof Object) ? ExtraOptions.AdditionalHeaders : {}); //Make sure extra options has an additional headers object
		ExtraOptions.AdditionalHeaders.Cookie=GetCookiesFromURL(CookieDomains); //Get cookies for the domains
	}
	
	return GetURL(URL, Data, ExtraOptions); //Do the AJAX Call
}

Since my FileSync Project is still a long way to being where I want it to be and is in a state that makes it annoying to use, I decided to throw together a script that essentially emulates the primary functions of it for what I need. I find it a bit annoying that I’ve never been able to find another good project that does exactly what I want for quick syncing of files over networks :-\. I used to think rsync would be a good solution for it but it’s very... quirky and unstable in certain ways. Not as flexible as I would like. Alas.

Anywho, this PHP script takes 2 file lists and gives you back their differences in a directory tree view. Each file has a data string after it used to tell if the files are different. The data string can be anything you want, but will usually probably be a timestamp or data checksum. The tree view lets you hide files/directories depending on whether each item is only on one side, different, or the same. An example is as follows (List1, List2):

Legend:

• Differences:
  • No differences detected
  • File is different, or directory contains differences
  • File is only found on left side, when this happens to a directory, it is only counted as 1 difference
  • File is only found on right side, when this happens to a directory, it is only counted as 1 difference
• Type:
  • This is a directory with sub items. After the directory name, it lists as info the number of differences over the total number of sub items
  • This is a file. If both sides contain the file but are different, it lists as info the different strings
• Info about the file/directory listed in between parenthesis.

Options:

• Display all items
• Display only same files (Displays all directories)
• Display different items (Does not display synced directories)
  • Do not display left only items
  • Do not display right only items

Left Side: ./Dir1.txt
Right Side: ./Dir2.txt
Total Differences: 9
Total Items: 13

• DiffDir (1/1)
  • Diff.txt (9999-99-99 99:99:99 :: 1111-11-11 11:11:11)
• LeftOnlyDir (1/2)
  • Left.txt
  • Left2.txt
• PartialDir (3/4)
  • Diff.txt (9999-99-99 99:99:99 :: 1111-11-11 11:11:11)
  • LeftOnly.txt
  • RightOnly.txt
  • Same.txt
• RightOnlyDir (1/1)
  • Right.txt
• SameDir (0/1)
  • Same.txt
• Diff.txt (9999-99-99 99:99:99 :: 1111-11-11 11:11:11)
• LeftOnly.txt
• RightOnly.txt
• Same.txt

• Output all files and their timestamps to “Dir1.txt”: find -type f -printf '%P\t%T+\n' > Dir1.txt
• Output all files and their md5sums to “Dir1.txt”: find -type f -print0 | xargs -0 md5sum | perl -pe 's/^(.*?) (.*)$/$2\t$1/g' > Dir1.txt

After discovering the Alamo Draft House’s coolness a few months ago, I’ve been trying to watch what they’re playing to make sure I catch anything I might want to see on the big screen. Unfortunately, it is not easy to get a good quick idea of all the movies playing from their calendar because it shows movies per day with showtimes, making the list repetitive and crowded with extra information.

I decided to throw together a real quick PHP script that would parse their data so I could organize it however I wanted. The final result can be viewed here. The code is as follows:

//The list of calendar pages in format TheaterName=>URL
$PagesToGrab=Array(
	'Ritz'=>'http://www.originalalamo.com/Calendar.aspx?l=2',
	'Village'=>'http://www.originalalamo.com/Calendar.aspx?l=3',
	'South Lamar'=>'http://www.originalalamo.com/Calendar.aspx?l=4'
);

foreach($PagesToGrab as $Name => $URL) //Grab the movies for each theater
{
	print "<b>$Name</b><br>"; //Output the theater name
	$TheHTML=file_get_contents($URL); //Grab the HTML
	$ShowList=Array(); //This will contain the final list of shows and what days they are on
	
	preg_match_all('/<td class="day">.*?<\/td>/', $TheHTML, $DayMatches); //Extract all table cells containing a day's info
	foreach($DayMatches[0] as $DayInfo) //Loop over each day's info
	{
		//Determine the day of month
		preg_match('/<a class=\"daynumber\" title=".*?, (.*?),/', $DayInfo, $DayOfMonth);
		$DayOfMonth=$DayOfMonth[1];
		
		//Determine all the shows for the day
		preg_match_all('/<span class="show"><a href=".*?">(.*?)<\/a>/', $DayInfo, $AllShows);
		foreach($AllShows[1] as $Show)
		{
			$Show=preg_replace('/^\s+|\s+$/', '', $Show); //Remove start and end of line whitespace
			if(!isset($ShowList[$Show])) //If show has not yet been added to overall list, add it
				$ShowList[$Show]=Array();
			$ShowList[$Show][]=$DayOfMonth; //Add this day as a time for the show
		}
	}
	
	//Output the shows and their days
	print '<table>';
	foreach($ShowList as $ShowName => $Days)
		print "<tr><td>$ShowName</td><td>".implode(', ', $Days).'</td></tr>';
	print '</table><br><br>';
}	
<? PageFooter(); ?>
</body></html>

<?
//This all works fine
print 'test1'_start;
require('foo/bar/test2.php');
print 'test1_end';
?>

<?
print 'test2_start';
require('blah/test3.php'); //This works fine because the include file it is in a subdirectory, not a parent directory of test2.php
require('../test4.php'); //This does not call fine (an error is thrown by php) because it is in a parent directory relative to test2.php, which was already included from the parent file (test1.php) in another directory (/). To fix this, use 'foo/test4.php'
print 'test2_end';
?>

function _require($IncludeFile, $CurrentFile)
{
	$CurrentPath=preg_replace('/[^\/]+$/', '', $CurrentFile); //Remove the filename to get the current path
	require(realpath("$CurrentPath$IncludeFile"));
}

I’ve been meaning to get searching working on my site for what seems like forever, and I decided to finally get around to getting some manner of search working via the temporary “use Google” solution. Unfortunately, it seems Google no longer does boolean searches completely properly as advertised. I am sure Google Search still supports boolean logic (as opposed to the assumed “and” between each word) because the Advanced Search, linked to from their front page, still has it, and it returns a few of the results it should.

(site:www.castledragmire.com/Projects OR site:www.castledragmire.com/Updates) AND (Fractal OR Font)

1. site:www.castledragmire.com/Projects Fractal
   • Projects
   • Projects/Fractal_Landscape
   • Projects/Julia_Fractals
   • Projects/Eyes
   • Projects/Games
2. site:www.castledragmire.com/Projects Font
   • Projects
   • Projects/ViewFonts
3. site:www.castledragmire.com/Updates Fractal
   • Updates/Fractal_Landscape_v0.5_and_v1.0
   • Updates/Fractal_Landscape_Technical_Paper_v1.0
   • Updates/Fractal_Landscape_v1.05
   • Updates/Julia_Fractals_Initial_Release
4. site:www.castledragmire.com/Updates Font
   • Updates/Site_Color_Scheme

• Projects
• Projects/Fractal_Landscape
• Projects/ViewFonts
  
  

Fractal OR Font site:www.castledragmire.com/Projects [Advanced Search]

• Projects
• Projects/Fractal_Landscape
• Projects/ViewFonts

Because of this, I need to go ahead and get real searching up via MySQL (or possibly another solution), as originally planned, sooner than later, since Google will not work as a temporary solution for what I want.

I wrote up a paper on what could be done through Google Search over 5 years ago as a job request [to be posted soon], which I believe is very informative. I’m sure it’s a little outdated, but it shows how much can Google can [could] do for you.

I’ve been incredibly busy lately, especially with work, but I finally have some time for personal stuff like posting again, yay. I’m currently stuck at the airport, and am leaving at 7AM this morning on vacation for 10 days on a tour of the west coast (Los Angeles, Disney World, Hollywood, Las Vegas, Grand Canyon, etc). The main reason for this get away is I’ll be meeting up with a good friend and his fiancée for their vacation and will be attending his wedding in Las Vegas ^_^.

I am currently on one of those open network connections at the airport that you have to pay to use, tunneled through one of my SSH servers, so I can bypassing their pay service and get online for free to post this :-). Hey, it’s their own fault for not securing it properly lol. I periodically kept getting dropped connections due to a weak signal, so I had to get back up after finding the connection and walk around, using my iPod to detect signal strengths until I found a better area with a stronger signal because. The thing is proving to be very useful ^_^. Anywho, on to the content of the post.

I’ve recently run into a number of new bugs [new to me at least] in both IE (version 7) and Firefox (version 3) that I have not encountered before and, as usual, have to program around to accomplish my tasks. I thought I’d discuss 3 of these bugs.

• Relative (non absolute) base URL paths do not work in either Firefox or Internet Explorer.
  

  Setting a base path for a website is often a necessity for websites that have web pages that are in subdirectories beyond the websites’s root directory. The reason for this is that the page-common layout of a web page usually refers to all images and content in a relative path. This is done for multiple reasons including:

  • Ease of moving the site between addresses like for test stages, or if the site is served from multiple domain names.
  • It’s easier to read source code URLs this way
  • It makes the HTML files smaller; though this isn’t a problem for most users these days because internet connection speeds are much faster.
  
  An example of W3C valid code that produces this error is as follows:

  <head><base href="/MySite/">
  

  The code, unfortunately, has to be an absolute URL like the following for current versions of IE and Firefox.

  <head><base href="http://domain.com/MySite/">
  

  
  

  One simple method to solve this problem is to use JavaScript to set an absolute base URL. Unfortunately, this then requires web browsers to have JavaScript enabled to work :-\. For this reason, this is really a quick fix for internal use that shouldn’t be put into production use unless JavaScript is required anyways.

  The following code will set a base of “http://domain.com/MySite/” for “http://domain.com/MySite/Posts/Post1.html”.

  <head>
  	<script type="text/javascript">
  		function GetBase() //Get the directory above the current path’s URL
  		{
  			return document.location.protocol+	//The protocol ("http:" or "https:")
  				'//'+				//End the protocol section with a //
  				document.location.hostname+	//The host (domain)
  				document.location.pathname.replace(/(\/[^/]*){2}$/,'')+ //This moves up 1 directory from the current path. To move up more directories, set the "2" in this line to NumberOfDirectoriesToMoveUp+1
  				'/';				//Add a '/' to set the end of the path as a directory
  		}
  		document.write('<base href="'+GetBase()+'">'); //Write a BASE object to set the current web page’s base URL to GetBase()
  	</script>
  </head>
  

  
  

  A simpler solution is to just have your parsing language (PHP for example) detect the server you are running on and set the proper base URL accordingly. This method assumes you know all the possible places/addresses your website will run on.

  <head><base href="<?=($_SERVER['HTTP_HOST']=='domain.com' ? 'http://domain.com/MySite/' : 'http://domain2.com/')?>"></head>
  

• Reserved keywords in IE cannot be used as object members
  Example (JavaScript):

  var MyObject={};
  MyObject.return=function() { return true; }
  

  Solution: Instances of this must be encoded in strings

  var MyObject={};
  MyObject['return']=function() { return true; }
  

  This also occurs for other reserved keywords like “debugger” and “for”.
• IE’s window does not have the “hasOwnProperty” member function like ALL OTHER OBJECTS
  

  This is a major nuisance because trying to find out if a variable exists and is not a prototype in the global scope is an important function. *sighs*

  The fix for this is using “window.VARIABLE!==undefined”, though this won’t tell you if the variable is actually instanced or [again] if it is part of the prototype; only if it is defined.

One more JavaScript engine difference between IE and Firefox is that in IE you can’t end a hash with an empty member. For example, the following works in Firefox, but not IE:

var b={a:1, b:2, c:3, d:4, };

This shouldn’t really be done anyways, so it’s not really a problem IMO. I ran across this when converting some bad Perl code (generated by YACC) which coincidentally allows this.

It’s really hard making everything compatible across all web browser platforms when they all contain so many nuances and bugs :-\.

I recently had to do some work on a system where I was not allowed SSH/telnet access. Trying to do work strictly across ftp can take hours, especially when you have thousands of files to transfer, so I came up with a quick solution in PHP for simple command line access.

<form method=post action="exec.php">
	<table style="height:100%;width:100%">
		<tr><td height="10%">
			<textarea name=MyAction style="height:100%;width:100%"><?=(isset($_REQUEST['MyAction']) ? $_REQUEST['MyAction'] : '')?></textarea>
		</td></tr><tr><td height="90%">
			<textarea name=Output style="height:100%;width:100%"><?
if(isset($_REQUEST['MyAction']))
{
	$MyAction=preg_split("/\\r?\\n/", $_REQUEST['MyAction']);
	foreach($MyAction as $Action)
	{
		exec($Action, $MyOutput);
		print htmlentities(implode("\n", $MyOutput), ENT_QUOTES, 'ISO8859-1')."\n-----------------------\n";
	}
}
			?></textarea>
		</td></tr><tr><td height=1>
			<input type=submit>
		</td></tr>
	</table>
</form>

This code allows you to enter commands on separate lines in the top box, and after the form is submitted, the output of each command is entered into the bottom box separated by dashed lines.

Note that between each command the environment is reset, so commands like "cd" which change the current directory are not useable :-(. You must also change the line 'action="exec.php"' to reflect the name you give the file.

A more suitable solution would be possible through AJAX and a program that redirected console output from a persistent session, but this was just meant as quick fix :-).

Since version 3 of Firefox, the browser has moved over from using flat files for keeping track of browsing history (history.dat) and bookmarks (bookmarks.html) to using SQLite databases (places.sqlite). This change over was required because the old flat file formats were badly implemented, clunky, and not able to handle the new demands of the location bar and browser history. Using a SQL database was the perfect solution for the complexity brought in with the new location bar and its dynamic searching of previous URLS, as SQL is easy to implement, is mostly compatible against multiple SQL application implementations (removing dependency on a single product), and powerful for cross referencing lookups. As a matter of fact, most of the data Firefox keeps now is stored in SQLite databases.

SQLite was also a good choice for the SQL solution because it can be implemented minimally straight into a product without needing a large install and a lot of bloat. While I like SQLite for this purpose and its ease of implementation, it lacks a lot of base SQL functionality that would be nice, like TABLE JOINS inside of DELETE statements, among many other language abilities. I wouldn’t suggest using it for large database driven products that require high optimization, which I believe it can’t handle. It’s meant as a simpler SQL implementation.

Anyways, I was very happy to see that when you delete URLs from the history in the newest version of Firefox that it actually deletes them out of the database as opposed to just hiding them, like it used to. The history manager actual seems to do its job quite well now, but I noticed one big problem. After attempting to delete all the URLs from a specific site out of the Firefox history manager, I noticed there were still some entries from that site in the SQLite database, which is a privacy problem.

After some digging, I realized that there are “hidden” entries inside of the history manager. A hidden entry is created when a URL is loaded in a frame or IFrame that you do not directly navigate too. These entries cannot be viewed through the history manager, and because of this, cannot be easily deleted outside of the history database without wiping the whole history.

At this point, I decided to go ahead and look at all the table structures for the history manager and figure out how they interact. Hidden entries are marked in places.sqlite::moz_places.history with the value “1”. According to a Firefox wiki “A hidden URL is one that the user did not specifically navigate to. These are commonly embedded pages, i-frames, RSS bookmarks and javascript calls.” So after figuring all of this out, I came up with some SQL commands to delete all hidden entries, which don’t really do anything anyways inside the database. Do note that Firefox has to be closed to work on the database so it is not locked.

sqlite3 places.sqlite
DELETE FROM moz_annos WHERE place_id IN (SELECT ID FROM moz_places WHERE hidden=1);
DELETE FROM moz_inputhistory WHERE place_id IN (SELECT ID FROM moz_places WHERE hidden=1);
DELETE FROM moz_historyvisits WHERE place_id IN (SELECT ID FROM moz_places WHERE hidden=1);
DELETE FROM moz_places WHERE hidden=1;
.exit

This could all be done in 1 SQL statement in MySQL, but again, SQLite is not as robust :-\. There is also a “Favorite’s Icon” table in the database that might keep an icon stored as long as a hidden entry for the domain still exists, but I didn’t really look into it.

I’ve been delving into the Perl language more lately for a job, and have found out some interesting things about it. Perl itself is a bit shrouded in mysticism, with it often being said that it runs on “magic”. The original Perl engine, written by Larry Wall, has never been duplicated due to its incredible complexity and hacked together nature.

@MyArray=('a',1,'b',2); #An array with values a,1,b,2
%MyHash=(a=>1, b=>2); #A hash with keys a,b that contain the values 1,2

@MyArray=('a'=>1=>'b'=>2); #An array with values a,1,b,2
%MyHash=(a,1,b,2); #A hash with keys a,b that contain the values 1,2

print $#MyArray; #Index of the last element, so add 1 to get length

$ArrayLength=@MyArray;
print $ArrayLength;

$MyRefArray=[1,2,3];
print scalar @$MyRefArray;
print $#$MyRefArray; #Index of the last element, so add 1 to get length

After now having delved a bit more into how Perl works, I still like PHP better as a strictly quick scripting language. Oh well.

JavaScript is a neat little scripting language and does the job it is intended for very well. The prototype system is very useful too, but has one major drawback. First, however, a very quick primer on how objects are made in JavaScript and what prototyping is.

function FooBar(ExampleArgument)
{
	this.Member1=ExampleArgument;
	this.AnotherMember='Blah';
}
var MyObject=new FooBar(5);

FooBar.prototype.NewMember=7;
var SecondObject=new FooBar();

FooBar.prototype.NewMember=9;

The way to detect if an object has a member is to use the in function, and then to determine if the member is prototyped, the hasOwnProperty function is used. For example:

'NewMember' in MyObject;			//Returns true
MyObject.hasOwnProperty('NewMember');		//Returns false

'Member1' in MyObject;				//Returns true
MyObject.hasOwnProperty('Member1');		//Returns true

'UnknownMember' in MyObject;			//Returns false
MyObject.hasOwnProperty('UnknownMember');	//Returns false

for(var i in MyObject)
	console.log( i + '=' + MyObject[i].toString() ); //console.log is a function provided by FireBug for FireFox, and Google Chrome

Member1=5
AnotherMember=Blah
NewMember=9

for(var i in MyObject)
	if(MyObject.hasOwnProperty(i))
		console.log(i+'='+MyObject[i].toString());

Member1=5
AnotherMember=Blah

This isn’t too bad if you are using prototyping yourself on your objects, but sometimes you might make objects that you wouldn’t expect to have prototypes. For good coding practice, you should really do the prototype check for every foreach loop because you can never assume that someone else will not add a prototype to an object type, even if your object type is private. This is especially true because all objects inherit from the actual Object object including its prototypes. So if someone does the following, which is considered very bad practice, every foreach loop will pick up this added member for all objects.

Object.prototype.GlobalMember=10;

Object.prototype.indexOf=function(Value)
{
	for(var i in this)
		if(this.hasOwnProperty(i) && this[i]===Value)
			return i;
	return undefined;
}

It would be really nice if “for(x in y)” only returned non-prototype members and there was another type of foreach loop like “for(x inall y)” that also returned prototype members :-\.

for(var i=0;i<ArrayObject.length;i++)

for(var i in ArrayObject)

//Array.indexOf prototype
if(Array.prototype.indexOf==undefined)
{
	function ArrayIndexOf(SearchIndex)
	{
		for(var i=0;i<this.length;i++)
			if(this[i]==SearchIndex)
				return i;
		return -1;
	}
	Array.prototype.indexOf=ArrayIndexOf;
}

I’m not going to go into how JavaScript stores the prototypes or how to find out all prototype members of an object, as that is a bit beyond what I wanted to talk about in this post, and it’s pretty self explanatory if you think about it.

function ClearCookies() //Clear all the cookies on the current website
{
	var MyCookies=document.cookie; //Remember the original cookie string since it will be changing soon
	var StartAt=0; //The current string pointer in MyCookies
	do //Loop through all cookies
	{
		var CookieName=MyCookies.substring(StartAt, MyCookies.indexOf('=', StartAt)).replace(/^ /,''); //Get the next cookie name in the list, and strip off leading white space
		document.cookie=CookieName+"=;expires=Thu, 01-Jan-1970 00:00:01 GMT; path=/"; //Erase the cookie
		StartAt=MyCookies.indexOf(';', StartAt)+1; //Move the string pointer to the end of the current cookie
	} while(StartAt!=0)
}

<input type=button value="Clear Cookies" onclick="ClearCookies()">
<input type=button value="View Cookies" onclick="ViewCookies()">
<textarea id=CookieBox style="width:100%;height:100%"></textarea>
<script type="text/javascript">
function ViewCookies() //Output the current cookies in the textbox
{
	document.getElementById('CookieBox').value=document.cookie.replace(/;/g,';\n\n');
}

function ClearCookies() //Clear all the cookies on the current website
{
	var CookieNames=[]; //Remember the cookie names as we erase them for later output
	var MyCookies=document.cookie; //Remember the original cookie string since it will be changing soon
	var StartAt=0; //The current string pointer in MyCookies
	do //Loop through all cookies
	{
		var CookieName=MyCookies.substring(StartAt, MyCookies.indexOf('=', StartAt)).replace(/^ /,''); //Get the next cookie name in the list, and strip off leading white space
		CookieNames.push(CookieName); //Remember the cookie name
		document.cookie=CookieName+"=;expires=Thu, 01-Jan-1970 00:00:01 GMT; path=/"; //Erase the cookie
		StartAt=MyCookies.indexOf(';', StartAt)+1; //Move the string pointer to the end of the current cookie
	} while(StartAt!=0)
	document.getElementById('CookieBox').value='Clearing: '+CookieNames.join("\nClearing: "); //Output the erased cookie names
}
</script>

One of the most laughable aspects of client/server* systems is client side based security access restrictions. What I mean by this is when credentials and actions are not checked and restricted on the server side of the equation, only on the client side, which can ALWAYS be bypassed.

To briefly explain why it is basically insane to trust a client computer; ANY multimedia, software, data, etc that has touched a person’s computer is essentially now their property. Once something has been on or through a person’s computer, the user can make copies, modify it, and do whatever the heck they want with it. This is how the digital world works. There are ways to help stop copying and modification, like hashes and encryption, but most of the ways in which things are implemented nowadays are quite fallible. There may be, for example, safeguards in place to only allow a user to use a piece of software on one certain computer or for a certain amount of time (DRM [Digital Rights Management]), but these methods are ALWAYS bypassable. The only true security comes by not letting information which people aren’t supposed to have access to cross through their computer, and keeping track of all verifiable factual information on secure servers. A long time ago at an IGDA [International Game Developers Association] meeting (I only ever went to the one unfortunately :-\), I learned an interesting truth that hadn’t occurred to me before from the lecturer. That is, that companies that make games and other software [usually] know it will sooner or later be pirated/cracked**. The true intention of software DRM is to make it hard enough to crack to discourage the crackers into giving up, and to make it take long enough so that hopefully people stop waiting for a free copy and go ahead and buy it. By the time a piece of software is cracked (if it takes as long as they hope), the companies know the majority of the remainder of the people usually wouldn’t have bought it anyways. Now I’m done with the basic explanation of client side insecurities, back to the real reason for this post.

While it is actually proper to program safeguards into client side software, you can never rely on it for true security. Security measures should always be duplicated in both client and server software. There are two reasons off the top of my head for implementing security access restrictions into the client side of software. The first is to help remove strain on servers. There is no point in asking a server if something is valid when the client can immediately confirm that it isn’t. The second reason is for speed. It’s MUCH quicker if a client can detect a problem and instantly inform the user than having to wait for a server to answer, though this time is usually imperceptible to the user, it can really add up.

So I thought I’d give a couple of examples of this to help you understand more where I’m coming from. This is a very big problem in the software industry. I find exploitable instances of this kind of thing on a very regular basis. However, I generally don’t take advantage of such holes, and try to inform the companies/programmers if they’ll listen. The term for this is white hat hacking, as opposed to black hat.

First, a very basic example. Let’s say you have a folder on your website “/PersonalPictures” that you wanted to restrict access to with a password. The proper way to do it would be to restrict access to the whole folder and all files in it on the server side, requiring a password be sent to the server to view the contents of each file. This is normally done through Apache httpd (the most utilized web server software) with an “.htaccess” file and the mod_auth (authentication) module. The improper way to do it would be a page that forwarded to the “hidden” section with a JavaScript script like the following.

if(prompt('Please enter the password')=='SecretPassword')
	document.location.href='/PersonalPictures';

The problem with this code is two fold (besides the fact it pops up a request window :-) ). First, the password is exposed in plain text to the user. Fortunately, passwords are usually not as easy to find as this, but I have found passwords in web pages and Flash code before with some digging (yes, Flash files (and Java!) are 100% decompilable to their original source code, sans comments). The second problem is that once the person goes to the URL “/PersonalPictures”, they can get back there and to all files inside it without the password, and also give it freely to others (no need to mention the fact that the URL is written in plain text here, as it’s the same as with the password). This specific problem with JavaScript was much more prevalent in the old day when people ran their web pages through free hosting sites like Geocities (now owned and operated by Yahoo) which didn’t allow for proper password protection.

This kind of problem is still around on the web, though it morphed with the times into a new form. Many server side scripts I have found across the Internet assume their client side web pages can take care of security and ignore the necessary checks in the server scripts. For example, very recently I was on a website that only allowed me to add a few items to a list. The way it was done is that there was a form with a textbox that you submitted every time you wanted to add an entry to the list. After submitting, the page was reloaded with the updated list. After you added the maximum allowed number of items to the list, when the page refreshed, the form to add more was gone. This is incredibly easy to bypass however. The normal way to do this would be to just send the modified packets directly to the server with whatever information you want in it. The easier method would be to make your own form submission page and just submit to the proper URL all you want. The Firebug extension for Firefox however makes this kind of thing INCREDIBLY easy. All that needs to be done is to add an attribute to the form to send the requests to a new window “<form action=... method=... target=_blank>”, so the form is never erased/overwritten and you can keep sending requests all you want. Using Firebug, you can also edit the values of hidden input boxes for this kind of thing.

AJAX (Asynchronous JavaScript and XML - A tool used in web programming to send and receive data from a server without having to refresh a page) has often been lampooned as insecure for this kind of reason. In reality, the medium itself is not insecure at all; it’s just how people use it.

As a matter of fact, the majority of my best and most fun Ragnarok hacking was done with these methods. I just monitored the packets that came in and out of the system, reverse engineered how they were all structured, then made modifications and resent them myself to see what I could do. With this, I was able to do things like (These should be most of the exploits; listed in descending order of usefulness & severity):

• Duplicate items
• Crash the server (It was never fixed AFAIK, but I stopped playing 5+ years ago. I just put that it was fixed on my site so people wouldn’t look for it ^_^; )
• Warp to any map from any warp location (warp locations are only supposed to link to 1 other map)
• Spoof your name during chats (so you could pretend someone else was saying something - Ender’s game, anyone? ^_^)
• Use certain skills of other classes (I have up pictures of my swordsman using merchant skills to house a selling shop)
• Add skills points to an item on your skill tree that is not yet available (and use it immediately)
• Warp back to save point without dying
• Talk to NPCs on a map from any location on that map, and sometimes from other maps (great for selling items when in a dungeon)
• Attack with weapons much quicker than was supposed to be allowed
• Use certain skills on creatures from any location on a map no matter how far they are
• Equip any item in any spot (so you could equip body armor on your head slot and get much more free armor defense points)
• Run commands on your party/guild and in chat rooms as if you were the leader/admin
• Rollback a characters stat’s to when you logged on that session (part of the dupe hack)
• Bypass text repetition, length, and curse filters
• Find out user account names

The original list is here; it should contain most of what I found. I took it down very soon after putting it up (replacement here) because I didn’t want to explicitly screw the game over with people finding out about these hacks (I had a lot of bad encounters with the company that ran the game, they refused to acknowledge or fix existing bugs when I reported them). There were so many things the server didn’t check just because the client wasn’t allowed to do them naturally.

Here are some very old news stories I saved up for when I wrote about this subject:

• [School board] “Google hacked our website”. I can’t find the original CNN article though.
• informit.com has another good example. Read the 2 paragraphs under restaurants.com under the table. The sheer stupidity never ceases to amaze me.

Just because you don’t give someone a way to do something doesn’t mean they won’t find a way.

According to a Microsoft KB article titled “Virtual directory names with executable extensions are not used correctly”, using a virtual folder ending in an executable extension (like .com, .exe, .dll, or .sh) under the web server for IIS [Microsoft’s Internet information services server suite] makes the contents inside the folder unviewable. This behavior itself is kind of silly, as you’d assume a web server would always check to see if something was a file or folder first.

Unfortunately, this doesn’t apply to just virtual folders, but all folders under an IIS web server, as I found out a few years ago when I backed up a site that I knew would be taken down very soon (ironically, because the company [SysInternals] was being taken over by Microsoft) and mirrored it on my Home Server, which runs IIS.

The solution I used was to add a character (in my case an underscore “_”) to the end of all the directory names ending in “.com” and then doing a global regular expression replace through all files in the mirror to replace any occurrences of these directories.

I still plan on getting up some site mirrors of places that no longer exist and such for the miscellaneous section one of these days...

A very important part of the design world is fonts, but it is an unfortunately annoying part of web browser land. There are very few fonts that come by default with OSs and even less default ones that match each other across all OSs, so your website won’t look the same across all platforms unless you use the right combinations. It’s much pretty guaranteed that if you want anything even remotely special in terms of a font somewhere on your website, you will be out of luck to match it across all platforms.

The commonplace solution for this is, of course, creating images for whenever you need special fonts displayed. While this is the most elegant solution, it is only appropriate for special circumstances, and not normal site content, as image file sizes can get ridiculous, and you lose plain text advantages like searchability and search engine recognition. Another solution is to request the user to download the font, like here. While this is a valid solution, the vast majority of users would not download the font because, mostly, they don’t care enough, and secondly, people generally know not to go download unfamiliar files on the internet when they don’t have to, for security reasons.

This has actually been a problem for me recently as I realized some of the default fonts I use for my site, which have always come with Windows, do not have default equivalents that come with most Linux distributions, as I had assumed. That’s a topic for a different day though.

So I had a customer recently request the ability to dynamically display some text in a certain font, so I told him there are 2 solutions. The first would be to use JavaScript to load translucent PNG images, the second would be to embed a Flash applet, as Flash can store font files internally for use. So here are instructions and examples of both:

• To start off for both paths, a screenshot (ALT+PRINT SCREEN to take only the current window) will need to be taken of the font rendered in black against a white background. This can be done in your favorite word processor as long as it properly renders with translucency, or (for Windows) by just going to the font file in “c:\windows\fonts” and opening it, which uses “fontview.exe”.
• After you have the screenshot, open a new file in Photoshop (File > New OR CTRL+N) and paste the screenshot into a new layer (Edit > Paste OR CTRL+V)
• Delete the background layer, which requires the layer window is open (Window > Layers OR F7 to toggle its display). Right click the text portion “Background” of the background layer, and choose “Delete Layer”.
• Select the region that contains your font’s alphabet (M for selection tool) and crop it (Image > Crop).
• You might want to zoom in at this point for easier viewing (CTRL++ for in, CTRL+- for out).
• The easy way from there:
  • Deselect the area (Select > Deselect OR CTRL+D).
  • Select the Magic Wand tool (W), set Tolerance to 0, check Anti-Aliased, and uncheck Contiguous
  • Select a pure white pixel and then delete the selection (DELETE)
  • You now have a translucent image that you can save and use, but the translucency isn’t that of the original font, as that is not how the magic wand tool works.
  Example using “Aeolus True Type Font” (Set against a green background via HTML for example sake)
  
• The better way:
  • Add a mask to your current layer (Layer > Add Layer Mask > Reveal All)
  • Go to the channels window (Window > Channels to toggle its display, it should be in the same window as Layers, in a separate tab) and select either the red, green, or blue layer. It doesn’t matter which as they should all hold the exact same values (grayscale [white-black colors] have the same red, green, and blue values), so red channel (CTRL+1) is fine.
  • Copy the channel (CTRL+C) (the entire workspace should still be selected after the crop)
  • Select the mask channel (CTRL+\), and you also need to make it visible (toggle the little eyeball icon besides it)
  • Paste into the mask channel (CTRL+V), invert it (Image > Adjustments > Invert OR CTRL+I), and then make it invisible again (untoggle little eyeball icon besides it)
  • Reselect the RGB contents (CTRL+~) and flood fill it with black [or your color of choice]: Paint Bucket Tool (G), 255 tolerance, no antialias
  • You now have a translucent image of the font that you can save and use that has the original font quality. You can test it by adding a white layer below it.
  Example using “Aeolus True Type Font” (Set against a green background via HTML for example sake)
  

• In a new Flash document (v5.0+), create a text box with the following properties:
  • Type: “Dynamic Text”
  • var: MyText
  • Font: YOURFONTCHOICE
  • Embed (button): Select the set of characters the dynamic text box might display. The less glyphs you select, the smaller the output file will be. I included all alpha-numeric+punctuation in the below example (24.3KB).
• That’s all you need for the Flash file, so all that’s left now is the JavaScript. The following function will set the text for you inside the movie. Also, you should set the embed (for normal browsers) and object (for IE) tags as different “id”s. The wmode is an important parameter here too, in that it makes the background invisible and the Flash applet more a part of the web page (not a “separate window”).

  <object width="300" height="40" id="CustomFontIE" classid="clsid:d27cdb6e-ae6d-11cf-96b8-444553540000">
  	<param name="movie" value="OtherContent/CustomFonts/CustomFont.swf">
  	<param wmode="transparent">
  	<embed src="OtherContent/CustomFonts/CustomFont.swf" wmode="transparent" width="300" height="40" id="CustomFont" type="application/x-shockwave-flash">
  </object>
  <script type="text/javascript">
  	var IsIE=(navigator.appName.indexOf('Microsoft')!=-1);
  	function SetFlashText(NewText) { document.getElementById('CustomFont'+(IsIE ? 'IE' : '')).SetVariable('MyText', NewText); }
  </script>
  		

So for reasons I’m not going to go into, today I had to compare some log files. I was tempted to write the code in C, just because I miss it so much these days x.x;, but laziness won out, especially as there weren’t that many log files and they weren’t that large, so I wrote it in PHP.

Nothing else to this post except the working code which took me about 5 minutes to type out... The function goes through one directory and all of its subdirectories and checks all files against the same path in a second directory. If the file doesn’t exist in the second directory or its contents doesn’t match the first file up to the first file’s length, a message is outputted.

//Start the log run against 2 root directories
TestLogs("/DIR1", "/DIR2");

function TestLogs($RootDir1, $RootDir2, $CurDir="")
{
	//Iterate through the first directory
	$Dir1=opendir("$RootDir1$CurDir");
	$SubDirs=Array(); //Holds subdirectories
	while($File=readdir($Dir1))
		if($File=="." || $File=="..") //Skip . and ..
			continue;
		else if(is_dir("$RootDir1$CurDir/$File")) //Do not try to compare directory entries
			$SubDirs[]=$File; //Remember subdirectories
		else if(!file_exists("$RootDir2$CurDir/$File"))
			print "File '$CurDir/$File' does not exist in second directory.<br>";
		else if(file_get_contents("$RootDir1$CurDir/$File")!=substr(file_get_contents("$RootDir2$CurDir/$File"),0,filesize("$RootDir1$CurDir/$File"))) //Both files exist, so compare them - if first file does not equal second file up to the same length, output error
			print "'$CurDir/$File' does not match.<br>";
	
	//Run subdirectories recursively after current directories' file-run so directories do not get split up
	foreach($SubDirs as $NewDir)
		TestLogs($RootDir1, $RootDir2, "$CurDir/$NewDir");
}

Today I thought I’d give a demonstration on the use of regular expressions [reference page here]. Regular expressions are basically a simplified scripting language for finding and replacing complex text strings, and are implemented into much of today’s software which involve a lot of text editing. They are a fabulously handy tool for computer users and are especially useful for programmers. I believe RegExps actually originally gained their notoriety through the Perl programming language. I also recently heard that it is definite that the new version of C++ (C++0x) will have native library support for regular expressions, yay!

Since I posted yesterday on DNS stuff, and have the examples from it handy, I figured I’d use those :-).

Let’s say you had a group of .com domains and wanted to find out their name servers (I’ve had to do this when switching to new name servers to make sure all the domains we did not control at the registrar level had their name servers set to the new ones). For this example, we will use the following domains “castledragmire.com”, “riaboy.com”, “NonExistantDomainA.com”, and “dakusan.com”.

• First, we’d need to have the list of the domains, for this example, one domain per line is used.
  castledragmire.com
  riaboy.com
  NonExistantDomainA.com
  dakusan.com
• Next, we need to turn them into a bash (Linux) script to grab all the information we need.
  Replace: “^(.*)$”
  With: “echo '!?$1?!'; host -t ns $1 a.gtld-servers.net | grep ' name server ';”
  Sample output: (The !? ?! stuff are markers for easier viewing and parsing)
  echo '!?castledragmire.com?!'; host -t ns castledragmire.com a.gtld-servers.net | grep ' name server ';
  echo '!?riaboy.com?!'; host -t ns riaboy.com a.gtld-servers.net | grep ' name server ';
  echo '!?NonExistantDomainA.com?!'; host -t ns NonExistantDomainA.com a.gtld-servers.net | grep ' name server ';
  echo '!?dakusan.com?!'; host -t ns dakusan.com a.gtld-servers.net | grep ' name server ';
• Next, we run the script, and it would output the following:
  !?castledragmire.com?!
  castledragmire.com name server ns3.deltaarc.com.
  castledragmire.com name server ns4.deltaarc.com.
  !?riaboy.com?!
  riaboy.com name server ns3.deltaarc.com.
  riaboy.com name server ns4.deltaarc.com.
  !?NonExistantDomainA.com?!
  !?dakusan.com?!
  dakusan.com name server ns3.deltaarc.com.
  dakusan.com name server ns4.deltaarc.com.
• Next, we would keep running the following regular expression until no more replacements are found.
  This would combine all domains with multiple name servers onto one line with name servers separated by spaces.
  Replace: “(.*?) name server (.*)\n\1 name server (.*)”
  With: “$1 name server $2 $3”
  It would output the following:
  !?castledragmire.com?!
  castledragmire.com name server ns3.deltaarc.com. ns4.deltaarc.com.
  !?riaboy.com?!
  riaboy.com name server ns3.deltaarc.com. ns4.deltaarc.com.
  !?NonExistantDomainA.com?!
  !?dakusan.com?!
  dakusan.com name server ns3.deltaarc.com. ns4.deltaarc.com.
• The final regular expression would turn the output into a single line per domain, followed by its domain servers. The current extra line before the list of name servers is to help spot any domains that did not provide us with name servers.
  Replace: “!\?(.*?)\?!\n\1 name server (.*)”
  With: “#$1 \t $2”
  Which would output the final following data:
  #castledragmire.com ns3.deltaarc.com. ns4.deltaarc.com.
  #riaboy.com ns3.deltaarc.com. ns4.deltaarc.com.
  !?NonExistantDomainA.com?!
  #dakusan.com ns3.deltaarc.com. ns4.deltaarc.com.
  This data could be directly pasted into Excel, which would put the first column as domains and second column as name servers).

Yesterday I wrote a bit about the DNS system being rather fussy, so I thought today I’d go a bit more into how DNS works, and some good tools for problem solving in this area.

• A network is simply a group of computers hooked together to communicate with each other. In the old days, all networking was done through physical wires (called the medium), but nowadays much of it is done through wireless connections. Wired networking is still required for the fastest communications, and is especially important for major backbones (the super highly utilized lines that connect networks together across the world).
• A LAN is a local network of all computers connected together in one physical location, whether it be a single room, a building, or a city. Technically, a LAN doesn’t have to be localized in one area, but it is preferred, and we will just assume it is so for arguments sake :-).
• A WAN is a Wide (Area) Network that connects multiple LANs together. This is what the Internet is.
• The way one computer finds another computer on a network is through its IP Address [hereby referred to as IPs in this post only]. There are other protocols, but this (TCP/IP) is by far the most widely utilized and is the true backbone of the Internet. IPs are like a house’s address (123 Fake Street, Theoretical City, Made Up Country). To explain it in a very simplified manner (this isn’t even remotely accurate, as networking is a complicated topic, but this is a good generalization), IPs have 4 sections of numbers ranging from 0-255 (1 byte). For example, 67.45.32.28 is a (class 4) IP. Each number in that address is a broader location, so the “28” is like a street address, “32” is the street, “45” is the city, and “67” is the country. When you send a packet from your computer, it goes to your local (street) router which then passes it to the city router and so on until it reaches its destination. If you are in the same city as the final destination of the packet, then it wouldn’t have to go to the country level.
• The final important part of networking (for this post) is the domain system (DNS) itself. A domain is a label for an IP Address, like calling “1600 Pennsylvania Avenue” as “The White House”. As an example, “www.castledragmire.com” just maps to my web server at “209.85.115.128” (this is the current IP, it will change if the site is ever moved to a new server).

• The root DNS servers (a.root-servers.net through m.root-servers.net) point to the servers that hold top-level-domain information (.com, .org., .net, .jp, etc)
  Examples of these servers are as follows:

  au	ns1.audns.net.au
  biz	E.GTLD.biz
  ca	CA04.CIRA.ca
  cn	A.DNS.cn
  com&net	A.GTLD-SERVERS.NET
  de	Z.NIC.de
  eu	U.NIC.eu
  info	B9.INFO.AFILIAS-NST.ORG
  org	TLD1.ULTRADNS.NET
  tv	C5.NSTLD.COM

• Next, these root name servers (like A.GTLD-SERVERS.NET through M.GTLD-SERVERS.NET for .com) hold two main pieces of information for ALL domains under their top-level-domain jurisdiction:
  • The registrar where the domain was registered
  • The name server(s) that are responsible for the domain
  Only registrars can talk to these root servers, so you have to go through the registrar to change the name server information.
• The final lowest rung in the DNS hierarchy is name servers. Name servers hold all the actual addressing information for a domain and can be run by anyone. The 2 most important (or maybe relevant is a better word...) types of DNS records are:
  • A: There should be many of these, each pointing a domain or subdomain (castledragmire.com, www.castledragmire.com, info.castledragmire.com, ...) to a specific IP address (version 4)
  • SOA: Start of Authority - There is only one of these records per domain, and it specifies authoritative information including the primary name server, the domain administrator’s email, the domain serial number, and several timeout values relating to refreshing domain information.

Now that we have all the basics down, on to the actual reason for this post. It’s really a nuisance trying to explain to people why their domain isn’t working, or is pointing to the wrong place. So here’s why it happens!

Back in the old days, it often took days for DNS propagation to happen after you made changes at your registrar or elsewhere, but fortunately, this problem is of the past. The reason for this is that ISPs and/or routers cached domain lookups and only refreshed them according to the metrics in the SOA record mentioned above, as they were supposed to. This was done for network speed reasons, as I believe older OSs might not have cached domains (wild speculation), and ISPs didn’t want to look up the address for a domain every time it was requested. Now, though, I rarely see caching on any level except at the local computer; not only on the OS level, but even some programs cache domains, like FireFox.

So the answer for when a person is getting the wrong address for a domain, and you know it is set correctly, is usually to just reboot. Clearing the DNS cache works too (for the OS level), but explaining how to do that is harder than saying “just reboot” ^_^;.

To clear the DNS cache in XP, enter the following into your “run” menu or in the command prompt: “ipconfig /flushdns”. This does not ALWAYS work, but it should work.

If your domain is still resolving to the wrong address when you ping it after your DNS cache is cleared, the next step is to see what name servers are being used for the information. You can do a whois on your domain to get the information directly form the registrar who controls the domain, but be careful where you do this as you never know what people are doing with the information. For a quick and secure whois, you can use “whois” from your linux command line, which I have patched through to a web script here. This script gives both normal and extended information, FYI.

Whois just tells you the name servers that you SHOULD be contacting, it doesn’t mean these are the ones you are asking, as the root DNS servers may not have updated the information yet. This is where our command line programs come into play.

In XP, you can use “nslookup -query=hinfo DOMAINNAME” and “nslookup -query=soa DOMAINNAME” to get a domain’s name servers, and then “nslookup NAMESERVER DOMAINNAME” to get the IP the name server points too. For example: (Important information in the following examples are bolded and in white)

C:\>nslookup -query=hinfo castledragmire.com
Server:  dns-redirect-lb-01.texas.rr.com
Address:  24.93.41.127

castledragmire.com
        primary name server = ns3.deltaarc.com
        responsible mail addr = admins.deltaarc.net
        serial  = 2007022713
        refresh = 14400 (4 hours)
        retry   = 7200 (2 hours)
        expire  = 3600000 (41 days 16 hours)
        default TTL = 86400 (1 day)

C:\>nslookup -query=soa castledragmire.com
Server:  dns-redirect-lb-01.texas.rr.com
Address:  24.93.41.127

Non-authoritative answer:
castledragmire.com
        primary name server = ns3.deltaarc.com
        responsible mail addr = admins.deltaarc.net
        serial  = 2007022713
        refresh = 14400 (4 hours)
        retry   = 7200 (2 hours)
        expire  = 3600000 (41 days 16 hours)
        default TTL = 86400 (1 day)

castledragmire.com      nameserver = ns4.deltaarc.com
castledragmire.com      nameserver = ns3.deltaarc.com
ns3.deltaarc.com        internet address = 216.127.92.71

C:\>nslookup ns3.deltaarc.com castledragmire.com
Server:  ev1s-209-85-115-128.theplanet.com
Address:  209.85.115.128

Name:    ns3.deltaarc.com
Address:  216.127.92.71

Nslookup is also available in Linux, but Linux has a better tool for this, as nslookup itself doesn’t always seem to give the correct answers, for some reason. So I recommend you use dig if you have it or Linux available to you. So with dig, we just start at the root name servers and work our way up to the SOA name server to get the real information of where the domain is resolving to and why.

root@www [~]# dig @a.root-servers.net castledragmire.com

; <<>> DiG 9.2.4 <<>> @a.root-servers.net castledragmire.com
; (2 servers found)
;; global options:  printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 5587
;; flags: qr rd; QUERY: 1, ANSWER: 0, AUTHORITY: 13, ADDITIONAL: 14

;; QUESTION SECTION:
;castledragmire.com.            IN      A

;; AUTHORITY SECTION:
com.                    172800  IN      NS      H.GTLD-SERVERS.NET.
com.                    172800  IN      NS      I.GTLD-SERVERS.NET.
com.                    172800  IN      NS      J.GTLD-SERVERS.NET.
com.                    172800  IN      NS      K.GTLD-SERVERS.NET.
com.                    172800  IN      NS      L.GTLD-SERVERS.NET.
com.                    172800  IN      NS      M.GTLD-SERVERS.NET.
com.                    172800  IN      NS      A.GTLD-SERVERS.NET.
com.                    172800  IN      NS      B.GTLD-SERVERS.NET.
com.                    172800  IN      NS      C.GTLD-SERVERS.NET.
com.                    172800  IN      NS      D.GTLD-SERVERS.NET.
com.                    172800  IN      NS      E.GTLD-SERVERS.NET.
com.                    172800  IN      NS      F.GTLD-SERVERS.NET.
com.                    172800  IN      NS      G.GTLD-SERVERS.NET.

;; ADDITIONAL SECTION:
A.GTLD-SERVERS.NET.     172800  IN      A       192.5.6.30
A.GTLD-SERVERS.NET.     172800  IN      AAAA    2001:503:a83e::2:30
B.GTLD-SERVERS.NET.     172800  IN      A       192.33.14.30
B.GTLD-SERVERS.NET.     172800  IN      AAAA    2001:503:231d::2:30
C.GTLD-SERVERS.NET.     172800  IN      A       192.26.92.30
D.GTLD-SERVERS.NET.     172800  IN      A       192.31.80.30
E.GTLD-SERVERS.NET.     172800  IN      A       192.12.94.30
F.GTLD-SERVERS.NET.     172800  IN      A       192.35.51.30
G.GTLD-SERVERS.NET.     172800  IN      A       192.42.93.30
H.GTLD-SERVERS.NET.     172800  IN      A       192.54.112.30
I.GTLD-SERVERS.NET.     172800  IN      A       192.43.172.30
J.GTLD-SERVERS.NET.     172800  IN      A       192.48.79.30
K.GTLD-SERVERS.NET.     172800  IN      A       192.52.178.30
L.GTLD-SERVERS.NET.     172800  IN      A       192.41.162.30

;; Query time: 240 msec
;; SERVER: 198.41.0.4#53(198.41.0.4)
;; WHEN: Sat Aug 23 04:15:28 2008
;; MSG SIZE  rcvd: 508

root@www [~]# dig @a.gtld-servers.net castledragmire.com

; <<>> DiG 9.2.4 <<>> @a.gtld-servers.net castledragmire.com
; (2 servers found)
;; global options:  printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 35586
;; flags: qr rd; QUERY: 1, ANSWER: 0, AUTHORITY: 2, ADDITIONAL: 2

;; QUESTION SECTION:
;castledragmire.com.            IN      A

;; AUTHORITY SECTION:
castledragmire.com.     172800  IN      NS      ns3.deltaarc.com.
castledragmire.com.     172800  IN      NS      ns4.deltaarc.com.

;; ADDITIONAL SECTION:
ns3.deltaarc.com.       172800  IN      A       216.127.92.71
ns4.deltaarc.com.       172800  IN      A       209.85.115.181

;; Query time: 58 msec
;; SERVER: 192.5.6.30#53(192.5.6.30)
;; WHEN: Sat Aug 23 04:15:42 2008
;; MSG SIZE  rcvd: 113

root@www [~]# dig @ns3.deltaarc.com castledragmire.com

; <<>> DiG 9.2.4 <<>> @ns3.deltaarc.com castledragmire.com
; (1 server found)
;; global options:  printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 26198
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 2, ADDITIONAL: 0

;; QUESTION SECTION:
;castledragmire.com.            IN      A

;; ANSWER SECTION:
castledragmire.com.     14400   IN      A       209.85.115.128

;; AUTHORITY SECTION:
castledragmire.com.     14400   IN      NS      ns4.deltaarc.com.
castledragmire.com.     14400   IN      NS      ns3.deltaarc.com.

;; Query time: 1 msec
;; SERVER: 216.127.92.71#53(216.127.92.71)
;; WHEN: Sat Aug 23 04:15:52 2008
;; MSG SIZE  rcvd: 97

Linux also has the “host” command, but I prefer and recommend “dig”.

And that’s how you diagnose DNS problems! :-). For reference, two common DNS configuration problems are not having your SOA and NS records properly set for the domain on your name server.

I also went ahead and added dig to the “Useful Bash commands and scripts” post.

I’m gonna cheat today since it is really late, as I spent a good amount of time organizing the 3D Engines update which pushed me a bit behind, and I’m also exhausted. Instead of writing some more content, I’m just linking to the “Utilized Optimization Techniques” section of the 3D Engines project, which I put up today.

It describes 4 programming speed optimization tricks: Local variable assignment, precalculating index lookups, pointer transversing/addition, and loop unrolling. This project post also goes into some differences between the used languages [Flash, C++, and Java], especially when dealing with speed.

I am often asked to transfer data sets into MySQL databases, or other formats. In this case, I’ll use a Microsoft Excel file without line breaks in the fields to MySQL as an example. While there are many programs out there to do this kind of thing, this method doesn’t take too long and is a good example use of regular expressions.

First, select all the data in Excel (ctrl+a) and copy (ctrl+c) it to a text editor with regular expression support. I recommend EditPad Pro as a very versatile and powerful text editor.

Next, we need to turn each row into the format “('FIELD1','FIELD2','FIELD3',...),”. Four regular expressions are needed to format the data:

• Add the start of the query: “INSERT INTO TABLENAME VALUES”
• End the query by changing the last row's comma “,” at the very end of the line to a semi-colon “;”.

a	b	c
d	e	f
g	h	i

INSERT INTO MyTable VALUES
('a','b','c'),
('d','e','f'),
('h','h','i');

Sometimes queries may get too long and you will need to separate them by performing the “2 more steps to complete the query” from above.

After doing one of these conversions recently, I was also asked to make the data searchable, so I made a very simple PHP script for this.

This script lets you search through all the fields and lists all matches. The fields are listed on the 2nd line in an array as "SQL_FieldName"=>"Viewable Name". If the “Viewable Name” contains a pound sign “#” it is matched exactly, otherwise, only part of the search string needs to be found.

<?
$Fields=Array('ClientNumber'=>'Client #', 'FirstName'=>'First Name', 'LastName'=>'Last Name', ...); //Field list
print '<form method=post action=index.php><table>'; //Form action needs to point to the current file
foreach($Fields as $Name => $Value) //Output search text boxes
	print "<tr><td>$Value</td><td><input name=\"$Name\" style='width:200px;' value=\"".
		(isset($_POST[$Name]) ? htmlentities($_POST[$Name], ENT_QUOTES) : '').'"></td></tr>';//Text boxes w/ POSTed values,if set
print '</table><input type=submit value=Search></form>';

if(!isset($_POST[key($Fields)])) //If search data has not been POSTed, stop here
	return;
	
$SearchArray=Array('1=1'); //Search parameters are stored here. 1=1 is passed in case no POSTed search parameter are ...
                           //... requested so there is at least 1 WHERE parameter, and is optimized out with the MySQL preprocessor anyways.
foreach($Fields as $Name => $Value) //Check each POSTed search parameter
	if(trim($_POST[$Name])!='') //If the POSTed search parameter is empty, do not use it as a search parameter
	{
		$V=mysql_escape_string($_POST[$Name]); //Prepare for SQL insertion
		$SearchArray[]=$Name.(strpos($Value, '#')===FALSE ? " LIKE '%$V%'" : "='$V'"); //Pound sign in the Viewable Name=exact ...
			//... value, otherwise, just a partial patch
	}
//Get data from MySQL
mysql_connect('SQL_HOST', 'SQL_USERNAME', 'SQL_PASSWORD');
mysql_select_db('SQL_DATABASE');
$q=mysql_query('SELECT * FROM TABLENAME WHERE '.implode(' AND ', $SearchArray));

//Output retrieved data
$i=0;
while($d=mysql_fetch_assoc($q)) //Iterate through found rows
{
	if(!($i++)) //If this is the first row found, output header
	{
		print '<table border=1 cellpadding=0 cellspacing=0><tr><td>Num</td>'; //Start table and output first column header (row #)
		foreach($Fields as $Name => $Value) //Output the rest of the column headers (Viewable Names)
			print "<td>$Value</td>";
		print '</tr>'; //Finish header row
	}
	print '<tr bgcolor='.($i&1 ? 'white' : 'gray')."><td>$i</td>"; //Start the data field's row. Row's colors are alternating white and gray.
	foreach($Fields as $Name => $Value) //Output row data
		print '<td>'.$d[$Name].'</td>';
	print '</tr>'; //End data row
}

print ($i==0 ? 'No records found.' : '</table>'); //If no records are found, output an error message, otherwise, end the data table
?>

I was thinking earlier today how it would be neat for C/C++ to be able to get the address of a jump-to label to be used in jump tables, specifically, for an emulator. A number of seconds after I did a Google query, I found out it is possible in gcc (the open source native Linux compiler) through the “label value operator” “&&”. I am crushed that MSVC doesn’t have native support for such a concept :-(.

The reason it would be great for an emulator is for emulating the CPU, in which, usually, each first byte of a CPU instruction’s opcode [see ASM] gives what the instruction is supposed to do. An example to explain the usefulness of a jump table is as follows:

void DoOpcode(int OpcodeNumber, ...)
{
	void *Opcodes[]={&&ADD, &&SUB, &&JUMP, &&MUL}; //assuming ADD=opcode 0 and so forth
	goto *Opcodes[OpcodeNumber];
  	ADD:
		//...
	SUB:
		//...
	JUMP:
		//...
	MUL:
		//...
}

Of course, this could still be done with virtual functions, function pointers, or a switch statement, but those are theoretically much slower. Having them in separate functions would also remove the possibility of local variables.

Although, again, theoretically, it wouldn’t be too bad to use, I believe, the _fastcall function calling convention with function pointers, and modern compilers SHOULD translate switches to jump tables in an instance like this, but modern compilers are so obfuscated you never know what they are really doing.

It would probably be best to try and code such an instance so that all 3 methods (function pointers, switch statement, jump table) could be utilized through compiler definitions, and then profile for whichever method is fastest and supported.

//Define the switch for which type of opcode picker we want
#define UseSwitchStatement
//#define UseJumpTable
//#define UseFunctionPointers

//Defines for how each opcode picker acts
#if defined(UseSwitchStatement)
	#define OPCODE(o) case OP_##o:
#elif defined(UseJumpTable)
	#define OPCODE(o) o:
	#define GET_OPCODE(o) &&o
#elif defined(UseFunctionPointers)
	#define OPCODE(o) void Opcode_##o()
	#define GET_OPCODE(o) (void*)&Opcode_##o
	//The above GET_OPCODE is actually a problem since the opcode functions aren't listed until after their ...
	//address is requested, but there are a couple of ways around that I'm not going to worry about going into here.
#endif

enum {OP_ADD=0, OP_SUB}; //assuming ADD=opcode 0 and so forth
void DoOpcode(int OpcodeNumber, ...)
{
	#ifndef UseSwitchStatement //If using JumpTable or FunctionPointers we need an array of the opcode jump locations
		void *Opcodes[]={GET_OPCODE(ADD), GET_OPCODE(SUB)}; //assuming ADD=opcode 0 and so forth
	#endif
	#if defined(UseSwitchStatement)
		switch(OpcodeNumber) { //Normal switch statement
	#elif defined(UseJumpTable)
		goto *Opcodes[OpcodeNumber]; //Jump to the proper label
	#elif defined(UseFunctionPointers)
		*(void(*)(void))Opcodes[OpcodeNumber]; //Jump to the proper function
		} //End the current function
	#endif

	//For testing under "UseFunctionPointers" (see GET_OPCODE comment under "defined(UseFunctionPointers)")
	//put the following OPCODE sections directly above this "DoOpcode" function
	OPCODE(ADD)
	{
		//...
	}
	OPCODE(SUB)
	{
		//...
	}

	#ifdef UseSwitchStatement //End the switch statement
	}
	#endif

#ifndef UseFunctionPointers //End the function
}
#endif

void ExamplePointerRetreival()
{
	void *LabelPointer;
	TheLabel:
	_asm mov LabelPointer, offset TheLabel
}

function ListContents($DirName)
{
	print '<ul>';
	$dir=opendir($DirName);
	while($file=readdir($dir))
		if($file!='.' && $file!='..')
		{
			$FilePath="$DirName/$file";
			$IsDir=is_dir($FilePath);
			print "<li>$file [".($IsDir ? 'D' : number_format(filesize($FilePath), 0, '.', ',')).']';
			if($IsDir)
				ListContents($FilePath);
			print '</li>';
		}
	closedir($dir);
	print '</ul>';
}

function HumanReadableSize($Size)
{
	$MetricSizes=Array('Bytes', 'KB', 'MB', 'GB', 'TB');
	for($SizeOn=0;$Size>=1024 && $SizeOn<count($MetricSizes)-1;$SizeOn++) //Loops until Size is < a binary thousand (1,024) or we have run out of listed Metric Sizes
		$Size/=1024;
	return preg_replace('/\\.?0+$/', '', number_format($Size, 2, '.', ',')).' '.$MetricSizes[$SizeOn]; //Forces to a maximum of 2 decimal places, adds comma at thousands place, appends metric size
}

function ListContents2($DirName, &$RetSize)
{
	$Output='<ul>';
	$dir=opendir($DirName);
	$TotalSize=0;
	while($file=readdir($dir))
		if($file!='.' && $file!='..')
		{
			$FilePath="$DirName/$file";
			if(is_dir($FilePath)) //Is directory
			{
				$DirContents=ListContents2($FilePath, $DirSize);
				$Output.="<li><b>$file</b> [".HumanReadableSize($DirSize)."]$DirContents</li>";
				$TotalSize+=$DirSize;
			}
			else //Is file
			{
				$FileSize=filesize($FilePath);
				$Output.="<li>$file [".HumanReadableSize($FileSize).']</li>';
				$TotalSize+=$FileSize;
			}
		}
	closedir($dir);
	$RetSize=$TotalSize;
	$Output.='</ul>';
	return $Output;
}

function CalcDirSize($DirName)
{
	$dir=opendir($DirName);
	$TotalSize=0;
	while($file=readdir($dir))
		if($file!='.' && $file!='..')
			$TotalSize+=(is_dir($FilePath="$DirName/$file") ? CalcDirSize($FilePath) :  filesize($FilePath));
	closedir($dir);
	return $TotalSize;
}

function ListContents3($DirName)
{
	print '<ul>';
	$dir=opendir($DirName);
	$TotalSize=0;
	while($file=readdir($dir))
		if($file!='.' && $file!='..')
		{
			$FilePath="$DirName/$file";
			$IsDir=is_dir($FilePath);
			$FileSize=($IsDir ? CalcDirSize($FilePath) : filesize($FilePath));
			$TotalSize+=$FileSize;
			print '<li>'.($IsDir ? '<b>' : '').$file.($IsDir ? '</b>' : '').' ['.HumanReadableSize($FileSize).']';
			if($IsDir) //Is directory
				$TotalSize+=ListContents3($FilePath);
			print '</li>';
		}
	closedir($dir);
	print '</ul>';
}

Find will search through a directory and its subdirectories for objects (files, directories, links, etc) satisfying its parameters.
Parameters are written like a math query, with parenthesis for order of operations (make sure to escape them with a “\”!), -a for boolean “and”, -o for boolean “or”, and ! for “not”. If neither -a or -o is specified, -a is assumed.
For example, to find all files that contain “conf” but do not contain “.bak” as the extension, OR are greater than 5MB:

find -type f \( \( -name "*conf*" ! -name "*.bak" \) -o -size +5120k \)

Some useful parameters include:

• -maxdepth & -mindepth: only look through certain levels of subdirectories
• -name: name of the object (-iname for case insensitive)
• -regex: name of object matches regular expression
• -size: size of object
• -type: type of object (block special, character special, directory, named pipe, regular file, symbolic link, socket, etc)
• -user & -group: object is owned by user/group
• -exec: exec a command on found objects
• -print0: output each object separated by a null terminator (great so other programs don’t get confused from white space characters)
• -printf: output specified information on each found object (see man file)

For any number operations, use:

+n		for greater than n
-n		for less than n
n		for exactly than n

For a complete reference, see your find’s man page.

xargs passes piped arguments to another command as trailing arguments.
For example, to list information on all files in a directory greater than 1MB: (Note this will not work with paths with spaces in them, use “find -print0” and “xargs -0” to fix this)

find -size +1024k | xargs ls -l

Some useful parameters include:

• -0: piped arguments are separated by null terminators
• -n: max arguments passed to each command
• -i: replaces “{}” with the piped argument(s)

So, for example, if you had 2 mirrored directories, and wanted to sync their modification timestamps:

cd /ORIGINAL_DIRECTORY
find -print0 | xargs -0 -i touch -m -r="{}" "/MIRROR_DIRECTORY/{}"

For a complete reference, see your xargs’s man page.

GREP is used to search through data for plain text, regular expression, or other pattern matches. You can use it to search through both pipes and files.
For example, to get your number of CPUs and their speeds:

cat /proc/cpuinfo | grep MHz

Some useful parameters include:

• -E: use extended regular expressions
• -P: use perl regular expression
• -l: output files with at least one match (-L for no matches)
• -o: show only the matching part of the line
• -r: recursively search through directories
• -v: invert to only output non-matching lines
• -Z: separates matches with null terminator

So, for example, to list all files under your current directory that contain “foo1”, “foo2”, or “bar”, you would use:

grep -rlE "foo(1|2)|bar"

For a complete reference, see your grep’s man page.

#!/bin/bash
export IFS=$'\n' #Forces only newlines to be considered argument separators
for dir in `find -type d -maxdepth 1`
do
	a=`find $dir -type f | wc -l`;
	if [ $a != "0" ]
	then
		echo $dir $a
	fi
done

<div id=example>
<script language=JavaScript>
	var MyObject=document.getElementById('example');
</script>

<body>
	<form name=MyForm>
		<input type=text name=ExampleText value=Example>
	</form>
	<script language=JavaScript>
		document.MyForm.ExampleText.value='New Example'; //Must use format document.FormName.ObjectName
	</script>
</body>