Git-annex notes and configuration

Table of Contents

Author Lee Hinman
Date 2016-08-02 16:59:26

Note that this document is a little outdated and is undergoing renovation as I get back into using git-annex.

1 Introduction

These are my notes when trying to use git-annex http://git-annex.branchable.com/.

Use at your own risk.

A word about names:

  • thulcandra is my local linux laptop running Fedora, where I do most of my development and browsing when on the road
  • ivalice-* is a Linux desktop running Fedora server in my house, where I do most development during the day if I'm at home
  • writequit is the machine that hosts http://writequit.org, I have it through a hosting provider, it's fairly powerful, and also hosts my Matrix homeserver, IRC bridge, etc
  • rpi is a raspberry pi that I run RetroPie on for retro/emulator gaming :)

1.1 Generating metadata for all files

I find it useful to set this up before committing files, since it uses a git-annex pre-commit to generate the metadata:

git config --global annex.genmetadata true

2 Documents

Usually this would be my computer's ~/Documents folder, but because a lot of programs decide they want to put random stuff in there (like Amazon MP3, Adobe stuff, Zoom, etc ಠ_ಠ), I created my own folder in ~/docs that I am using for storing actual documents.

3 Photos

For photos, I want to keep some locally, and all of them backed up on an external USB drive. I want git-annex to keep track of what is where, so I know where to go and get the file.

On the laptop side:

mkdir ~/pics
cd ~/pics
git init
git annex init "laptop"


mkdir /Volumes/MINIDRIVE1/pics
cd /Volumes/MINIDRIVE1/pics
git init
git annex init "usbdrive"
git remote add laptop ~/pics


cd ~/pics
git remote add usbdrive /Volumes/MINIDRIVE1/pics

... add files ...

git annex sync

And when I want to move them off of my local machine and only on the external drive:

cd ~/pics
git annex move <folder> --to usbdrive
git annex sync

On the USB drive side:

cd /Volumes/MINIDRIVE1/pics
git annex sync

Then, if I am running low on space, I can safely "drop" images and git-annex will ensure I still have at least one copy of the data:

cd ~/pics
git annex drop photos-from-trip

3.1 Custom tagged views

http://git-annex.branchable.com/tips/metadata_driven_views/

What I would like to do is to be able to view documents based on a time structure, instead of by category, so if I have:

$ tree -d ~/pics
/Users/hinmanm/pics
├── christmas-party-2011-12
└── christmas-party-2013-12

It would be nice if I could have:

$ tree -d ~/pics
/Users/hinmanm/pics
├── 2011
    └── christmas-party-2011-12
└── 2013
    └── christmas-party-2013-12

To tag a folder with a year tag:

git annex metadata --set year=2012 pics-from-2012-03
git annex metadata --set month=03 pics-from-2012-03
# or, with a random tag:
git annex metadata --tag europe pics-from-europe

And then check out the view:

git annex view year=* month=*

To do the tagging automatically, we can configure genmetadata:

git config annex.genmetadata true
# or, globally
git config --global annex.genmetadata true

4 Videos

I'd like to save some various shows, and presentations that people have given. I use the youtube-dl tool for downloading a most of the videos.

On my laptop side:

cd
mkdir videos
cd videos
git init
git annex init "laptop"
git remote add delta ssh://delta-local/mnt/data/annex/videos

... add files ...

git annex sync

On the file server side:

cd /mnt/data/annex
mkdir videos
cd videos
git init
git annex init "delta"
git remote add laptop ssh://xanadu/Users/hinmanm/videos

... add files ...

git annex sync

5 Podcasts

I recently switched to using git-annex to manage podcasts, so far I really like it.

Here's how I set it up:

mkdir ~/podcasts
cd ~/podcasts
git init
git annex init "laptop"
echo "http://theshipshow.com/podcast.xml" >> podcast-urls

Repeat for any podcast URL you feel appropriate. I then have two scripts:

get-podcasts.sh

#!/bin/sh
xargs git annex importfeed --template='${feedtitle}/${itempubdate}-${itemtitle}${extension}' < podcast-urls

And fast-get-podcasts.sh

#!/bin/sh
xargs git annex importfeed --fast --template='${feedtitle}/${itempubdate}-${itemtitle}${extension}' < podcast-urls

The only difference between the two is the --fast option, which means only the metadata is downloaded instead of all the files. I tend to use fast-get-podcasts.sh when I'm at a cafe or coffeeshop, and get-podcasts.sh when I'm at home.

When downloading all of the meta, I can get the actual file with git annex get <file>, like so:

cd ~/podcasts/The_Ship_Show/
git annex get "2015_04_21-Episode_55__I_Don_t_Always_Test__But_When_I_Do....mp3"

And then git-annex will download the file, when I'm done or don't want to keep the file around, I do

git annex drop "2015_04_21-Episode_55__I_Don_t_Always_Test__But_When_I_Do....mp3"

And the file is dropped from the local repo.

5.1 synchronize the files to phone

I have a separate podcast method for my phone, need to figure out how to sync the two…

6 Files

7 S3 backup

I would like to keep local and off-site (meaning s3) backups of my photos, using git-annex.

Author: Lee Hinman

Created: 2016-08-02 Tue 16:59

Validate