Creating 7zip files on Linux with LZMA

So today I was trying to create an archive that someone had specifically requested be in 7zip format from my Linux laptop and I ran in to a few annoying “features” of the very basic p7Zip application availible in most Debian/Ubuntu based systems.

p7Zip essentially only has two command line options, compress or decompress and nothing you can tell it will stop it from deleting the original source file once it’s finished compressing. This is worrying.

Another annoying feature is it’s choice to have no option to choose an output filename. Infact even the utility I’m going to introduce in a second doesn’t have this option either but it’s at least trivial to work around.

If you didn’t already know, 7zip is actually just an implementation of LZMA and LZMA2 compression with the file extention ‘.7z.’

Given this we can use the command lzma (sudo apt-get install lzma.)

Here is the command issued to create a 7zip a file with lzma and then a test decompress with p7zip to prove it works.

(~ $ lzma -kvzc file.tst > file.7z)

Now an explanation of the command one option at a time.

-k, this forces lzma to keep the original file and not delete it once the archive is complete.

-v, verbose output so we can see what’s going on.

-z, this is the option that tells lzma that we want to create an archive.

And finally the most interesting bit;

-c, this option forces lzma to output the file not to a file but to the stdout, this can then be redirected to a file of our choosing using the “> file.7z”. Replacing the filename with whatever you want.

New Linux Kernel: Couldn’t find PV pv n. Check your device.map

When upgrading to a new kernel on Debian this error can present itself if you have previously added an extra drive to your LVM pool without having updated the /boot/grub/device.map.

The simplest way to get round this is to backup your current device.map file, delete the original and then run..

$# sudo update-grub

followed by trying to run the kernel install again.

If this hasn’t solved your issue you may have to add the new drive to the device.map file manualy

$# cat /boot/grub/device.map

(hd0) /dev/disk/by-id/ata-VBOX_HARDDISK_VBfdab3e2d-25850c00

(as you can see in this example only one drive is listed.)

$# sudo nano /boot/grub/device.map

<I would add the following line at the end of the file>

(hd1) /dev/device

(device can be eiher the standard /dev/sdxn or could be by id as per the first entry of the device.map file)

[ If this post helped you, please rate or leave a comment. ]

Enabling file ACL’s in Debian 6 (Access control lists)

I’ve been brushing up on my knowledge of Linux ACL’s this morning and thought I’d share some of the highlights with you.

Those of you with a Unix or Windows Server background will be well versed with the idea of access control lists on-top of basic file permissions but it seems to be something, which even today is less extensively used in the Linux world.

In a nutshell, for those unfamiliar with access control lists, they give you a finer granularity of control over  file or directory permissions than can be achieved with the standard Unix file permissions.

Take the following example for instance;

You have a number of users in you finance department who are all part of the ‘finance’ group, you also have a subset of these users who are involved with a special project and so are also members of the group ‘specialproject’. Out of those members of the ‘specialproject’ group, only John and Bob should have write access to the special project document but the others in the group should still be able to read the file.

With normal Unix permissions you could set the file to be owned by John and in the special project group with ‘rw’ permissions for the owner and just ‘r’ permissions for the group and your problem is solved for John and the special project group but not for Bob.

Now John and Bob could be added to yet another group and ‘rw’ permission  given to the file for that group but now as the file is owned by this new group instead of ‘specialgroup’ the rest of your special projects group have lost access.

This can be solved fairly trivially though by adding an ACL entry specifying Bob to have ‘rw’ access to the file which will over-ride the standard linux group permissions.

So, the standard Unix permissions would give John ‘rw’ access, the members of the ‘specialprojecct’ group ‘r’ access and then the ACL would specifically give Bob his ‘rw’ access to the file

(output of how the permissions would appear)

#$ ls -al

-rw-r—–+ 1 john specialproject 37 Feb 10 04:42 special_project_file

(notice the ‘+’ after the permissions, this specifies that an acl has been applied to this file)

(to check the acl entry you can do the following)

#$ getfacl /opt/t1/admin/finance_team/special_project_file
getfacl: Removing leading ‘/’ from absolute path names
# file: opt/t1/admin/finance_team/special_project_file
# owner: john
# group: specialproject
user::rw-
user:bob:rw-
group::r–
other::—

Despite ACL support being compiled in to both my custom kernel and the standard Debian version it was not enable at the filesystem level by default.

To do this you can enabled it until reboot/remount of your filesystem by running the following command.

#$ sudo mount -o remount,acl /

(replacing / with the filesystem which you wish to enable ACL’s for)

To enable this permanently you will need to edit you ‘/etc/fstab’ and add ‘acl’ to the list of options for each partitions it’s required on. In my case that meant changing the option ‘defaults’ to ‘acl,defaults’ for my ‘/’, ‘/home’, ‘/tmp’ directories, etc.

Now that your filesystem has ACL enabled you’ll need to grab the user-land tools to read and modify your ACL permissions.

#$ sudo apt-get install acl

This installs the following tools, ‘/bin/getfacl’, ‘/bin/setfacl’ and ‘/bin/chacl’ which can be used to manipulate and read back ACL settings. Information on using each of these can be looked up in ‘man’.

(a few examples of using these commands)

#$ setfacl -m user:john:rwx ./myfile (gives user john rwx access to myfile)

#$ setfacl -m group:admin:rw ./myfile (give the group admin rw access to myfile)

#$ getfacl ./myfile (display ACL entries for myfile)

 

(the following example creates a folder that will automatically propagates its acl permissions to all file contained within)

#$ mkdir shared_area

#$ setfacl -d -m u:john:7 shared_area

#$ setfacl -d -m u:bob:7 shared_area

#$ touch ./shared_area/testfile

#$ ls -al ./shared_area

drwxr-xr-x+ 2 john john 4096 Feb 10 14:08 .
drwxrwx— 6 john staff 4096 Feb 10 14:06 ..
-rw-rw-r–+ 1 john john 0 Feb 10 14:08 testfile

(and now to prove that the acl has been inherited to the new file we’ve just created)

#$ getfacl ./shared_area/testfile

# file: shared_area/testfile
# owner: john
# group: john
user::rw-
user:john:rwx #effective:rw-
user:bob:rwx #effective:rw-
group::r-x #effective:r–
mask::rw-
other::r–

Sucsessful upgrade from Debian 5 (lenny) to 6 (squeeze) and extending LVM volume with new Hard disk.

Well, last night I bravely attempted an upgrade of Debian lenny to squeeze which I would like to report overall was a breeze.

I won’t be posting a guide but I would like to point any readers attempting the same to a very useful blog post which helped me though the process.

Following the steps detailed here was a greate help.

I have to state that apart from a few packages installed from source that my Debian install wasn’t too far from the original, with the only external repository added to my source list being the tor repository located here, (deb http://deb.torproject.org/torproject.org lenny main), which for the purpose of the upgrade I comented out.

I don’t run debian natively, instead preferinmg to run it inside vmware on top of OS X with SSH and Xforwarding enabled to integrate the two nicely although I pretty much always have both running. I only have to sully myself with Windows while at work now thankfully.

Despite for the main part the upgrade going smoothly  I did run out of space on my root partition before the ‘apt-get dist upgrade’ stage.  Luckily apt flaged this before downloading the packages and due to VMware and LVM the proccess of adding extra space was not taxing.

I simply added a new SCSI drive from the VMWare settings, rebooted Debian to make sure it was seen, although I beleive even this step may have been unecisary due to the nature of SCSI in Linux.

Once rebooted I ssh’d back on to the machine, ran ‘su’ to give me root privileges and did the following;

Please note you could potentially loose all your data performing an operation like this. Always make sure to back up before doing something like this. I cannot guarantee that this will work. Despite having been successful for me any major alteration to your disk or partitions could have catastrophic effects.

debian:/# cfdisk /dev/sdc

(replace sdc with the appropriate name of your new disk, create a partition of type ‘Linux’ with the size you wish. In my case I used the entire disk )

debian:/# lvmdiskscan

(to check that lvm2 is seeing the disk and partition)

debian:/# pvcreate /dev/sdc1

(to add the partition to LVM)

debian:/# vgextend debnet /dev/sdc1

(debnet being the name of my LVM VG. Replace this with the correct name for your setup)

debian:/# lvm lvextend -l +100%FREE /dev/debnet/root (extend the LVM partition called ‘root’ to use 100% of free space)

debian:/# resize2fs -p /dev/mapper/debnet-root

(resize my ext3 partion contained in ‘debian-root’ to use the extra space)

That’s it. My root partition had now been extended to use the extra hard disk. LVM saves the day again and potentially this could have all been done without a reboot.

At this point I continued with the install and am now happily running Squeeze.

Installing exFat on Debian based distros

[Edited to update the download link – Aug 2012]

[Edit –  Included info on making .deb – Aug 2012]

ExFat, otherwise known as FAT64 is a new filesystem introduced by Microsoft with Windows Vista SP1.

The reasoning behind the new filesystem was to address an area in which Microsoft’s previous filesystems where not a neat fit, i.e. on large external flash drives where NTFS is not appropriate but FAT32 is not sufficient.

exFAT has a theoretical maximum disk size of 64 ZiB, 2,796,202 files per subdirectory and many other improvements.

Many modern SDXC cards are now coming formated with exFat and it was from one of these which I wanted to access from my Debian machine.

Linux support for exFAT is still very much experimental and Debian has no pre-made binary package yet although I didn’t have much trouble in compiling from source and installing.

As always when compiling from source you will need to have the GNU C compiler and associated tools installed.

If not already present you can do this as follows;

debian:/#su

debian:/#apt-get install build-esential libfuse-dev

Instead of the normal ‘make’ build system exFAT uses ‘Scons‘ so you may need to install this to.

debian:/#apt-get install scons

Once installed you can now download the exFAT source code from http://code.google.com/p/exfat/downloads/list

(Direct link to file as of writing, fuse-exfat-0.9.8.tar.gz)

Extract the tarball by typing;

debian:/#tar -xvf ./fuse-exfat-0.9.8.tar.gz

Navigate in to the directory containing the expanded files and run the following command to compile the source code and create a folder for the mount point. (You’ll need to have appropriate access permission to install, i.e. root )

debian:/#scons install

debian:/#sudo apt-get install checkinstall

debian:/#sudo checkinstall scons install

this will create a .deb file which can be installed and deleted with your package manager.

debian:/#sudo dpkg -i nameofpackage.deb

debian:/#mkidr /mnt/exfat

Now that exFAT is installed you can mount your external disk as follows;

debian:/#mount.exfat-fuse /dev/sdXn /mnt/exfat

You’ll need to replace the ‘X’ with the drive location and ‘n’ with the disk partition number, for example, ‘/dev/sdb5’.

Python on Debian from sourcecode update (altinstalling development branch 3.2 rc2)

Just though I’d post an update on my earlier post about installing Python 3.1.3 from source code on Debian Linux.

Although I wouldn’t encourage you to use it as your main environment some of you may be interested in trying out the latest development branch of Python to test new features or to help with finding and reporting bugs.

As of writing this, the lattest release is 3.2 rc2.

http://www.python.org/ftp/python/3.2/Python-3.2rc2.tgz

If you’d like to test out the development branch without interfering with your main install of Python you can use Pythons ‘altinstall’ feature.

Follow my previous guide on compiling Python 3.1.3 and when you reach the final stage instead of running;

debian:/#make && make install

Run the following instead;  (please note you will need appropriate permission to write to system folders to run either ‘make install’ or ‘make altinstall’.)

debian:/#make && make altinstall

Instead of installing the Python binary under the default name of  ‘python3’, ‘altinstall’ will incorporate the full version number in to the binary name leaving the original stable branch binary undisturbed.

You can run the ‘altinstall’ by typing;

debian:/#python3.2

Link to my previous guide:
Installing Python 3.1.3 from source, the full works (Debian Linux 5.x lenny)

Installing Python 3.1.3 from source, the full works (Debian Linux 5.x lenny)

So previously I’d posted about getting Python 3.1 installed from source on OS X. This time I’ll be talking about installing Python 3.1.3 on Debian Linux 5.x lenny.

Although this guide is specific to Debian lenny it should be applicable to other Linux distros’ based on Debian such as Ubuntu and can probably be adapted to work on other distros’.

First you’ll need to download the Python source code, you can do so by following this link. Python 3.1.3

You can also download and uncompress the file by running the following commands from the terminal

debian:/# wget http://www.python.org/ftp/python/3.1.3/Python-3.1.3.tar.bz2

debian:/# tar -xvf ./Python-3.1.3.tar.bz2

To build programs from source you’ll need to install gcc and associated files which can be done as follows;

debian:/# apt-get install build-essential

The first time I tried to compile the Python source-code I was presented with the following output….

Python build finished, but the necessary bits to build these modules were not found:
_curses            _curses_panel      _dbm
_gdbm              _hashlib           _sqlite3
_ssl               _tkinter           bz2
readline           zlib
To find the necessary bits, look in setup.py in detect_modules() for the module’s name.

I should expect on most fairly default installs of Debian you’ll also be missing the same components. If you wanted you could run Python as it is but you’ll be missing many important features.

If you follow the rest of this guide I’ll show you how to install all the required files for a fully featured Python build.

To install the missing components you should run the following from the terminal;

debian:/# apt-get install zlib1g-dev libncurses5-dev libreadline5-dev libssl-dev libbz2-dev libsqlite3-dev tk-dev libgdbm-dev

Now you have all the required components we can try to compile the code.

Navigate through to the directory where you extracted the Python files and run the following.

debian:/#./configure && make && make install

If everything has gone well Python 3.1.3 should now be compiled and installed.

you can run the Python interpreter by running the following;

debian:/#python3

I hope this guide has been helpful. Happy coding.

Wargames.

I’ve recently started playing one of the wargames called ‘IO’ hosted at www.smashthestack.org.


If you’ve not heard of them before they host a number of games to test your skills at computer hacking, programming and your in-depth knowledge of how computer systems work.

Each game is hosted on it own server which you access via ssh. At the begging you are provided with the username and password for level1. It is your task to complete the challenge set for you in that level. Once completed you will have escalated your user privileges to that of the subsequent level, allowing you to read a file in the next levels home folder containing it’s password. Once retrieved you can log out and back in with the next levels username and password and attempt the following challenge.
Although for a seasoned pro’ the first few levels of ‘IO’ are probably considered easy they  are still tough enough that significant knowledge is required to complete them.

The first level consists of a program which asks you to enter a password. Depending on your input this will then either output “Fail.” or “Win” along with a new shell running as level2 privileges.

The aim of this task is to analyse this executable  file and extract the password string contained within. I won’t reveal the solution but one approach would be to debug the program with gdb, find the code which does the string comparison between your entry and the correct password and then to read the password stored at the memory address. The other approach is much quicker but maybe slightly defeats the point of the level as an introduction to gdb.

The second level consists of a programming  and maths challenge. The level2 program asks for a password which consists of the results of a range of numbers in a specified series, joined together one after the other as a string.

#level2@io:~$ /levels/level02#Append the 39th through 42nd numbers in the sequence as a string and feed it to# this binary via argv[1]. 1, 2, 3, 5, 8, 13, 21…#The 4th through the 7th numbers would give you 581321

I completed this by creating a python script to create the series of numbers needed, concatenate the appropriate ones together in to a string and to give that as an argument to the level2 executable. Win.

For tasks like this you may find it quicker to do the codeing on your own machine and then upload to the io server. To do this you can use the ‘scp’ command from the terminal. (you can download scp for windows from the putty website).  First create a directory in the io server /tmp folder.

Give it a hard to guess unique name as although directory listing is restricted on the server and others can’t see your folder name if you use an easily guessable directory name others could still access it by chance.

While logged in to the io server type, ‘mkdir /tmp/mydirname’. Replacing mydirname with your unique directory name.

Then from the terminal on your machine you can upload files as such;

scp -P 2224 ~/files/level2.py level2@io.smashthestack.org:/tmp/mydirname/

The command is broken down as follows, ‘-P 2224’ specifies the port number as io does not run on the standard ssh port number, ‘~/files/level2.py’ is the location of my file to upload on my machine, ‘level2’ is the username with which I want to connect to the remote machine, io.smashthestack.org is the remote server address, the ‘:’ separates the remote server address from the final part which is the location on the remote machine to store my file.

I’ve just started the 3rd level and upon running the level3 program it instantly seg’ faults. Here in lies the challenge. Hopefully this should keep me occupied for a while.

To join in yourself connect to the io server like so;

ssh level1@io.smashthestack.org -p2224
password: level1

(linux and os x)

From windows you will need to download an ssh client like this… (putty)

Here is a snippet from the ./README file you can view when you connect.

Welcome to the IO wargame at the smash the stack network.

———————————————————
You have done the hard part and found our realm. Here we allow you to play with classic, and up to date vulnerabilities in software. Since many of you may be unfamiliar with how a wargame works, we will give a quick introduction in the following paragraphs. If you are an experienced wargamer, all this will be familiar to you so you might want to skip to the last section which iterates the specifics of this game.
The problems will be presented to you as a series of programs. Which will vary in size from a few lines containing an obvious bug, over to larger, and finally real software. The point is always to exploit this bug in such a way that you can grab control of the programs execution and make it do what you want. For example you will often want it to drop a shell.
The way this works is that the binaries are SUID binaries (http://en.wikipedia.org/wiki/Setuid). This means in short that they run as a different user than you do. The point is to grab control of the program and make it execute your own shellcode. Which will in turn allow you to read the password for the next level.

Once you have completed each level you can add your ‘tag’ to an html file which can be accessed from the wargame website. This is your proof to the world that you completed the level. As a word of warning do not view the tag page in your browser with JavaScript enabled. Remember this file could contain anything previous level winners have entered so it could have potentially dangerous or annoying consequences for anyone who fails to heed this warning.

If you’d like to see proof that I completed level 1 & 2 so far you can look for the tag ‘retrop’ in the respective tag pages. Once again do not click these links with JavaScript enabled. Level 2 tags (do not click without reading warning), Level 3 tags (Do not click before reading warning).